Hands-onIntermediate 35 min· Machine learning · 11 of 20

Feature engineering

Turn raw columns into features a model can use: dates, ratios, log transforms, missing values and categories. Build a ColumnTransformer pipeline and prove with cross-validation which new features help.

What you will do: take a raw table of customers, fix missing values, turn text categories into numbers, create new columns that carry more information, and measure whether each change helps. You will end with a reusable scikit-learn pipeline.

Prerequisites: pandas basics, the classification tutorial, and *Train, validation, test and overfitting* (so you know why preprocessing must be done inside a pipeline). Everything runs in Colab or Kaggle; the data is created by the code.

The plan

1Create the data and look at the problems in it.
2Make new columns from dates, ratios and transforms.
3Fill in missing values.
4Encode categories as numbers.
5Combine everything in a pipeline and train.
6Run an ablation to see which features helped.

What is a feature?

A feature is one input column the model sees. Feature engineering is choosing, cleaning and creating those columns so the model can find the pattern. On tabular data (rows and columns, like a spreadsheet) it is often worth more than switching to a fancier algorithm.

Models are not smart about raw data. A model cannot use the text "Bengaluru". It cannot use a date as a date. It cannot do arithmetic between columns unless you do it. Your knowledge of the problem (people who visit often come back; newer customers are less loyal) is what turns raw columns into useful ones.

The data

We create 600 customers of a coffee subscription. The task: predict whether a customer returned the next month. The data has text categories (city, plan), a date (signup_date), a skewed money column and missing values. The target was generated so that tenure, visits and premium plan matter.

Create the data

import numpy as np
import pandas as pd

rng = np.random.default_rng(7)
n = 600

df = pd.DataFrame({
    "customer_id": np.arange(1, n + 1),
    "city": rng.choice(["Bengaluru", "Pune", "Chennai", "Jaipur"], size=n, p=[0.4, 0.3, 0.2, 0.1]),
    "plan": rng.choice(["basic", "standard", "premium"], size=n, p=[0.5, 0.35, 0.15]),
    "signup_date": pd.to_datetime("2024-01-01") + pd.to_timedelta(rng.integers(0, 600, size=n), unit="D"),
    "monthly_spend": rng.lognormal(mean=6.0, sigma=0.7, size=n).round(0),
    "visits_last_30d": rng.poisson(6, size=n).astype(float),
})
# Some visits are unknown, as in real data
df.loc[rng.choice(n, size=40, replace=False), "visits_last_30d"] = np.nan

# The target: did the customer come back next month? (made up: loyal, frequent customers return more)
tenure = (pd.Timestamp("2025-12-31") - df["signup_date"]).dt.days
logit = -2.0 + 0.004 * tenure + 0.2 * (df["visits_last_30d"].fillna(6) - 6) + 0.8 * (df["plan"] == "premium")
df["returned"] = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)

pd.set_option("display.width", 120)
pd.set_option("display.max_columns", 20)
print(df.head())
print(df.isna().sum())
print("Share who returned:", round(df["returned"].mean(), 2))

Builds a DataFrame called df and prints a summary.

Output

   customer_id       city      plan signup_date  monthly_spend  visits_last_30d  returned
0            1       Pune   premium  2025-03-16          484.0              NaN         1
1            2    Chennai  standard  2024-02-20          105.0             10.0         1
2            3    Chennai     basic  2024-03-29          168.0              NaN         1
3            4  Bengaluru     basic  2025-03-03         1022.0              6.0         0
4            5  Bengaluru     basic  2024-01-21          209.0              7.0         1
...
visits_last_30d    40
...
Share who returned: 0.43

Step 1: create new columns

Four common ideas:

- Dates: a raw date is a single big number. Extract what matters: how long ago (tenure), the month (seasonality), the day of week. - Ratios: spend per visit says something that spend and visits alone do not. - Transforms: money, counts and times are often skewed (many small values, a few huge ones). log1p (log of 1 plus the value) compresses the tail. This helps linear models and neural networks, and makes no difference to trees. - Binning and flags: group a number into ranges, or add a yes/no column such as "visited at all".

Engineered columns

# New columns made from the ones we already have
snapshot = pd.Timestamp("2025-12-31")
df["tenure_days"] = (snapshot - df["signup_date"]).dt.days
df["signup_month"] = df["signup_date"].dt.month
df["signup_weekday"] = df["signup_date"].dt.day_of_week          # 0 = Monday
df["spend_per_visit"] = df["monthly_spend"] / (df["visits_last_30d"] + 1)  # +1 avoids dividing by zero
df["log_spend"] = np.log1p(df["monthly_spend"])               # tames a long right tail

print(df[["tenure_days", "signup_month", "spend_per_visit", "log_spend"]].head())

Needs the df from the first cell.

Output

   tenure_days  signup_month  spend_per_visit  log_spend
0          290             3              NaN   6.184149
1          680             2         9.545455   4.663439
2          642             3              NaN   5.129899
3          303             3       146.000000   6.930495
4          710             1        26.125000   5.347108

Notice the snapshot date. We measure tenure as of a fixed date, because in a real project tenure must be calculated as of the moment you make the prediction, never as of today. Using today's date would give different training and prediction values.

Step 2: handle missing values

Most scikit-learn models cannot accept NaN. Options:

- Drop rows with missing values. Simple, but you lose data and may bias the sample. - Impute: fill with a typical value. The median is safe for skewed numbers; the mean for symmetric ones; the most frequent value for categories. - Add a missing indicator: a 0/1 column saying "this value was missing". Sometimes the fact that a value is missing is itself informative.

SimpleImputer(strategy="median", add_indicator=True) does the last two together. The median is learned from the training data only, when it sits inside a pipeline.

Step 3: encode categories

Which encoder?

Type	Example	Encoder	Result
Nominal (no order)	city: Pune, Chennai	OneHotEncoder	One 0/1 column per city
Ordinal (has order)	plan: basic < standard < premium	OrdinalEncoder with the order given	0, 1, 2
High cardinality	pin code, 5000 different values	Frequency or target encoding, or a tree model that handles categories	One numeric column

Numbers that pretend to mean something

Do not label cities 0, 1, 2, 3 and feed the numbers to a linear model. It would think Jaipur (3) is "three times" Pune (1) and that Chennai sits between them. Use one-hot for categories with no natural order.

Step 4: put it all together in a pipeline

A ColumnTransformer applies a different preprocessing recipe to each group of columns, and a Pipeline chains it to the model. Then cross_val_score re-fits every step on each training fold, so nothing leaks.

Preprocessing recipe

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler

numeric_cols = ["monthly_spend", "visits_last_30d", "tenure_days", "spend_per_visit", "log_spend"]
nominal_cols = ["city"]          # no natural order
ordinal_cols = ["plan"]          # basic < standard < premium

numeric_pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median", add_indicator=True)),
    ("scale", StandardScaler()),
])

preprocess = ColumnTransformer([
    ("num", numeric_pipe, numeric_cols),
    ("city", OneHotEncoder(handle_unknown="ignore"), nominal_cols),
    ("plan", OrdinalEncoder(categories=[["basic", "standard", "premium"]]), ordinal_cols),
])

Defines numeric, nominal and ordinal handling. handle_unknown='ignore' stops the model crashing on a city it never saw.

Train and inspect

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split

feature_cols = numeric_cols + nominal_cols + ordinal_cols
X = df[feature_cols]
y = df["returned"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

model = Pipeline([
    ("prep", preprocess),
    ("clf", LogisticRegression(max_iter=1000)),
])

scores = cross_val_score(model, X_train, y_train, cv=5, scoring="accuracy")
print("CV accuracy:", scores.mean().round(3))

model.fit(X_train, y_train)
print("Test accuracy:", round(model.score(X_test, y_test), 3))

# What did the preprocessing produce?
names = model.named_steps["prep"].get_feature_names_out()
print(len(names), "columns")
print(list(names))

Builds the full model, cross-validates it and lists the columns the model actually receives.

Output

CV accuracy: 0.675
Test accuracy: 0.633
12 columns
['num__monthly_spend', 'num__visits_last_30d', 'num__tenure_days', 'num__spend_per_visit', 'num__log_spend', 'num__missingindicator_visits_last_30d', 'num__missingindicator_spend_per_visit', 'city__city_Bengaluru', 'city__city_Chennai', 'city__city_Jaipur', 'city__city_Pune', 'plan__plan']

Five numeric columns, two missing-indicator columns, four city columns and one plan column make 12. The accuracy is modest because the data is noisy by design. The test accuracy (0.633) is below the cross-validated one (0.675); with only 120 test rows, that gap is within normal variation.

Step 5: test whether the new features helped

Never assume a feature helps. Run an ablation: remove it (or the group) and compare cross-validated scores. Use the same folds and the same model.

Compare with and without engineered features

# Ablation: remove the engineered columns and compare
base_numeric = ["monthly_spend", "visits_last_30d"]
base_cols = base_numeric + nominal_cols + ordinal_cols

base_prep = ColumnTransformer([
    ("num", numeric_pipe, base_numeric),
    ("city", OneHotEncoder(handle_unknown="ignore"), nominal_cols),
    ("plan", OrdinalEncoder(categories=[["basic", "standard", "premium"]]), ordinal_cols),
])
base_model = Pipeline([("prep", base_prep), ("clf", LogisticRegression(max_iter=1000))])
print("Without new features:", cross_val_score(base_model, X_train[base_cols], y_train, cv=5).mean().round(3))
print("With new features:   ", cross_val_score(model, X_train, y_train, cv=5).mean().round(3))

The base model uses only the raw spend and visits columns, plus city and plan.

Output

Without new features: 0.592
With new features:    0.675

Adding tenure (from the signup date) lifted accuracy from about 0.59 to about 0.68. This is no accident: we built the data so tenure matters. In your own projects, the same experiment tells you honestly which ideas are worth keeping.

Good habits

Checklist

Build features from information available at prediction time only. A column that is created after the outcome is leakage.
Put imputers, scalers and encoders inside a Pipeline or ColumnTransformer.
Test one idea at a time with cross-validation, and keep the ones that help.
Drop ID columns such as customer_id before training; they carry no pattern and invite memorising.
Prefer features you can explain to a business person.
Remember that tree models need no scaling and cope with skew; linear models and neural networks benefit from both.

Prompt · Brainstorm features for my dataset

I am building a model to predict [target] for [business]. My table has these columns:

[column name: type and an example value, one per line]

Suggest 10 new features I could engineer from these columns (dates, ratios, counts, flags, transformations). For each, give a one-line reason it might help and the pandas code to create it. Flag any that might leak information not known at prediction time.

What to expect: A good answer gives specific, working pandas lines and warns about leakage. Treat the suggestions as hypotheses to test with cross-validation, not facts.

Try it: Add your own features

Add a feature is_weekend_signup (1 if the signup weekday is Saturday or Sunday), a column high_spender (1 if monthly_spend is above the 75th percentile) and a visited_often flag (visits at least 8). Add them to the pipeline and check cross-validated accuracy. Did anything help? What does it tell you if nothing helps?

Try it: Try one-hot versus ordinal for city

Replace the OneHotEncoder for city with an OrdinalEncoder and compare the score. Then replace LogisticRegression with RandomForestClassifier and repeat. When does it matter?

Key takeaways

Features are the columns the model sees. Good features often beat a fancier algorithm on tabular data.
Create features from dates (tenure, month, weekday), ratios, log transforms and flags.
Impute missing values (median is a safe default) and consider a missing-value indicator.
One-hot encode categories with no order; use an ordinal encoding only when there is a real order.
Put all preprocessing in a ColumnTransformer inside a Pipeline so cross-validation stays honest.
Test every new idea with an ablation using cross-validation, and keep only what helps.

Check yourself

Finishing the quiz marks the lesson as done.

1. Which encoding is best for a column 'city' with values Pune, Jaipur, Chennai when you use logistic regression?

2. Why should an imputer be placed inside a Pipeline rather than applied to the whole dataset first?

3. You are predicting churn as of 1 December. Which feature would be leakage?

4. What does np.log1p(x) help with?

5. How do you find out whether a new feature really helps?

Saved on this device. Create a free account to keep it on every device.

Metrics: accuracy, precision, recall, F1 and ROC Decision trees, random forests and gradient boosting

Practise this with Sam

A real mock interview, out loud, with an honest scorecard. Free to start.

Start free