Data Preprocessing: Imputation, Encoding, Scaling & Leakage Prevention

In textbook computer science, data structures are clean and deterministic. In real-world AI engineering, raw datasets arriving from production databases, web analytics, and CSV dumps are messy, incomplete, inconsistently scaled, and filled with arbitrary text strings.

Common Real-World Data Defects & Preprocessing Remedies

Every transformer in scikit-learn has internal parameters that are learned from data. For instance, StandardScaler learns the sample mean (μ) and standard deviation (σ); SimpleImputer learns median or mean statistics; and OneHotEncoder learns the list of observed categories.

# DANGEROUS LEAKAGE!
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X) # Test stats leak in!
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y)

# SAFE & LEAKAGE-FREE
X_train, X_test, y_train, y_test = train_test_split(X, y)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train) # Learn on train
X_test_scaled = scaler.transform(X_test)      # Apply to test

The Triad: fit(), transform(), and fit_transform()

Beginners often resort to df.dropna() to eliminate missing values. In production AI engineering, blindly dropping rows is dangerous: it destroys statistical power, biases the sample, and crashes production serving if live requests contain missing fields.

Scikit-Learn SimpleImputer Strategies

Categorical features contain discrete qualitative values. They fall into two distinct mathematical groups:

• Nominal: Categories with no inherent ordering (e.g. City: "Mumbai", "Delhi", "Pune"; or Operating System: "Linux", "macOS", "Windows").
• Ordinal: Categories with a natural, monotonic rank (e.g. Education: "High School" < "Bachelor's" < "Master's" < "PhD").

Scikit-Learn OneHotEncoder: Modern API (1.4+)

In current versions of scikit-learn, two critical parameters govern how OneHotEncoder operates:

from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder(
    sparse_output=False,       # Current API: returns dense 2D NumPy array (sparse=False is removed)
    handle_unknown='ignore'    # Production resilience: encode unseen categories as all zeros
)

# Learn categories from training data: ['Delhi', 'Mumbai', 'Pune']
X_train_cat = encoder.fit_transform(X_train[['city']])

# When test data contains 'Bangalore' (unseen), handle_unknown='ignore' creates [0, 0, 0] safely!
X_test_cat = encoder.transform(X_test[['city']])

Consider a dataset predicting loan approval with two features: Age (range 18 to 70) and Annual Income (range ₹25,000 to ₹1,500,000). If you compute Euclidean distance between two applicants, a difference of ₹10,000 in income completely dwarfs a 30-year difference in age by a factor of hundreds.

The Three Core Scalers Compared

An outlier is not automatically bad data. In fraud detection, high-net-worth customer analytics, or website latency monitoring, the outliers are often the most critical signals in the entire problem.

import numpy as np
from sklearn.preprocessing import PowerTransformer

# Method 'yeo-johnson' works with both positive and negative continuous values
pt = PowerTransformer(method='yeo-johnson', standardize=True)

# Compresses skewed distribution into normal bell curve
X_train_transformed = pt.fit_transform(X_train[['income']])
X_test_transformed = pt.transform(X_test[['income']])

Real enterprise data is virtually never all-numeric or all-categorical. A typical customer record contains numeric columns (age, monthly_usage) and categorical columns (city, plan_type).

In legacy workflows, data scientists manually sliced DataFrames, scaled numeric columns, one-hot encoded strings, and manually glued them back together with np.hstack(). This pattern was brittle, error-prone, and guaranteed data leakage in cross-validation.

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

num_cols = ['age', 'income', 'tickets']
cat_cols = ['city', 'plan']

preprocessor = ColumnTransformer(
    transformers=[
        ('num', Pipeline([
            ('imputer', SimpleImputer(strategy='median')),
            ('scaler', StandardScaler())
        ]), num_cols),
        ('cat', Pipeline([
            ('imputer', SimpleImputer(strategy='most_frequent')),
            ('onehot', OneHotEncoder(sparse_output=False, handle_unknown='ignore'))
        ]), cat_cols)
    ]
)

# Retain pandas DataFrame with readable feature names!
preprocessor.set_output(transform='pandas')

X_train_clean = preprocessor.fit_transform(X_train)
X_test_clean = preprocessor.transform(X_test)

The ultimate weapon against data leakage and training-serving skew is the scikit-learn Pipeline. Instead of maintaining independent preprocessor and classifier objects, you bind them into a single estimator:

from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

# Bundle Preprocessing AND Model together!
model_pipeline = Pipeline([
    ('preprocess', preprocessor),
    ('classifier', LogisticRegression(random_state=42))
])

# Single call fits preprocessor on X_train AND trains logistic regression!
model_pipeline.fit(X_train, y_train)

# Inference automatically passes raw records through preprocessor before predicting!
predictions = model_pipeline.predict(X_test)

In this project, you are given raw customer records containing missing values, categorical plan types, and disparate usage metrics. Step through the 4 production stages to build a leakage-free predictive pipeline.

Check off each core competency as you master it. Aim for 100% completion before building production ML models:

Many software engineers assume data preprocessing is an obsolete concern of 2015-era classical machine learning. In reality, the fundamental challenge of data representation and training-serving parity is even more vital in modern AI systems: