Better Workflow · Lesson 4 of 7

Pipelines

Keeping your steps clean

So far, preprocessing (imputing, encoding) and modelling are separate steps you run by hand. That gets messy and error prone, especially once you split data. A pipeline bundles all the steps into one object you fit and predict with as a unit. The benefits:

Cleaner code: you stop juggling training and validation data through every step by hand.
Fewer bugs: you cannot forget a step or apply it inconsistently between train and validation.
Easier to deploy: the whole preprocessing-plus-model bundle moves as one piece.
Better validation: cross-validation (next lesson) needs this to work correctly.

You build one in three steps: define the preprocessing, define the model, then bundle them.

A ColumnTransformer applies the right preprocessing to each kind of column (numeric vs categorical), and a Pipeline chains that to the model:

Python

import pandas as pd
import numpy as np
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error

students = pd.DataFrame({
    "study_hours":    [2, 5, 1, 6, 3, 8, 4, 7, 1, 6, 5, 3, 7, 4, 8],
    "attendance_pct": [60, 85, 50, 95, 70, 98, 75, 90, 45, 88, 80, 65, 92, 78, 99],
    "prev_score":     [50, 72, np.nan, 85, 58, 92, np.nan, 88, 35, 80, 75, 55, 84, 62, 95],
    "region":         ["Rural","Urban","Rural","Urban","Rural","Urban","Rural","Urban","Rural","Urban","Rural","Rural","Urban","Rural","Urban"],
    "exam_score":     [52, 78, 41, 90, 61, 96, 68, 89, 38, 82, 77, 59, 87, 66, 98],
})
X = students.drop(columns=["exam_score"])
y = students["exam_score"]

num_cols = ["study_hours", "attendance_pct", "prev_score"]
cat_cols = ["region"]

# Step 1: preprocessing for each kind of column
numeric = SimpleImputer(strategy="mean")              # numeric: fill gaps with the mean
categorical = Pipeline([                               # categorical: fill gaps, then one-hot encode
    ("impute", SimpleImputer(strategy="most_frequent")),
    ("onehot", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
])
pre = ColumnTransformer([
    ("num", numeric, num_cols),
    ("cat", categorical, cat_cols),
])

# Step 2 + 3: define the model and bundle everything into one pipeline
model = Pipeline([
    ("pre", pre),
    ("rf", RandomForestRegressor(random_state=1)),
])

train_X, val_X, train_y, val_y = train_test_split(X, y, random_state=1)
model.fit(train_X, train_y)                            # fit on RAW training data
print("Validation MAE:", round(mean_absolute_error(val_y, model.predict(val_X)), 1))  # predict on RAW validation data

The payoff

Notice two things. First, you fit the whole bundle in one line: the pipeline imputes, encodes, and trains the model together. Second, you call predict on the raw validation data, and the pipeline preprocesses it for you automatically, the exact same way it preprocessed training. You can never forget a step, mismatch the columns, or leak validation data into preprocessing. This is how real projects are built.

Practice

Run the cell. Then change the model from RandomForestRegressor to DecisionTreeRegressor(random_state=1) (remember to import it) and re-run. Did the MAE go up or down? Pipelines make swapping models this easy.

You’re reading for free. Sign in to keep your progress and earn a certificate when you finish.Sign in to keep my progress →

← PreviousCategorical variables Next →Cross-validation