Data Quality & Cleaning Missing Value Handling Pandas 2.x Verified Live Interactive Lab

Python Data Cleaning: Handling Missing Values

Master professional missing value detection, diagnosis, and remediation. Learn why missing data happens, how to measure missingness percentages, evaluate the trade-offs of dropping vs filling, and apply robust statistical imputation (mean, median, mode, constant) to build production-ready datasets.

Level: Beginner to Intermediate

Environment: Python 3 / Pandas 2.x

Est. Time: 50–65 Minutes

1. What are Missing Values?

Understanding unavailable information in enterprise records

In real-world data analytics, datasets are rarely complete. Consider this human resources log:

Name	Age	Salary ($)
Amit	24	45,000
Priya	NaN	50,000
Rahul	27	NaN
Neha	25	60,000

In Pandas, missing values appear as NaN (Not a Number) for numerical data or None. This indicates that information is unavailable or unrecorded.

Common Reasons Data is Missing:

User Skipped Field: Optional fields on sign-up forms (e.g. phone number or middle name).
Unrecorded / Technical Drop: IoT sensor disconnected, network dropped, or database timeout.
Not Applicable: A customer without a company car will have no "Car Mileage" data.
Data Extraction Bugs: Malformed JSON or CSV parsing errors during ETL pipelines.

⚠️ Critical Distinction: Missing data is NOT the same as 0 or "No"!
A bank balance of $0 means the account is empty (a legitimate number). A bank balance of NaN means we have no idea how much money the person has. Treating missing values as 0 distorts statistical analyses!

2. Finding Missing Values: isna() and notna()

Generating boolean masks and column summary counts

Pandas provides two primary methods to locate missing values:

df.isna()

Returns True if missing, False if present.

df.notna()

Returns True if valid data, False if missing.

df.isna().sum()

Sums Trues per column to give exact missing counts.

Why is df.isna().sum() so powerful?
In Python, boolean True evaluates as 1 and False evaluates as 0. Calling .sum() counts every missing value in each column instantly!

3. 🔥 Live Missing-Value Detection

Inspect nulls and generate summary statistics

Given the df below, practice detecting and counting missing values:

missing_detection.py

# Task 1: Find all missing values across the DataFrame

# Task 2: Count how many missing values exist in each column
# df.isna().sum()

# Task 3: Check which values are NOT missing
# df.notna()

Click "Run Python" to test your code...

4. Understand the Extent: Calculating % Missing

Relative percentages guide the cleaning strategy

A count of 50 missing values means something very different in a dataset of 100 rows (50% missing!) versus a dataset of 1,000,000 rows (0.005% missing). Analysts always calculate the percentage of missing values:

Percentage Calculation Blueprint

# Calculate percentage of missing values per column
missing_pct = (df.isna().sum() / len(df)) * 100
print(missing_pct)

The Analyst Golden Rule:
"Before touching or cleaning any dataset, always measure the exact extent of the missing data."

5. Dropping Missing Values: df.dropna()

Removing incomplete rows or empty columns

The fastest way to eliminate missing values is using dropna():

df.dropna() (Default: Rows)

Removes any row that contains at least one NaN.

df_clean = df.dropna()

df.dropna(axis=1) (Columns)

Removes any column that contains at least one NaN.

df_clean = df.dropna(axis=1)

6. When Should You Drop Data?

Never assume: Missing value = Delete row

Deleting records is a destructive operation. Consider the trade-offs:

Safe to Drop When

• Only a tiny fraction (< 1–2%) of records are incomplete.
• Critical identifier is missing (e.g. missing Customer_ID in transactions).
• The record is completely unrecoverable for the specific business question.

Dangerous to Drop When

• 15–40% of the rows would vanish.
• Sample size shrinks drastically, ruining statistical confidence.
• Introduces bias (e.g. low-income users skipping salary fields; dropping them skews average salary upward!).

7. 🔥 Live dropna() Practice

Observe how much data is removed across rows vs columns

dropna_lab.py

# Task 1: Remove all rows containing at least one missing value

Click "Run Python" to test dropna...

8. Filling Missing Values: df.fillna()

Preserving sample size through sensible imputation

Instead of deleting records, an analyst can replace missing values using fillna():

fillna Blueprint

# Replace nulls in a specific column with a value
df["Department"] = df["Department"].fillna("Unknown")

# Replace nulls with the column's median
df["Age"] = df["Age"].fillna(df["Age"].median())

⚠️ Analyst Warning: Filling with 0 is NOT automatically correct!
If you fill missing Age with 0, you introduce 0-year-old babies into an adult workforce, ruining the average age metric!

9. Common Imputation Strategies for Data Analysts

Choosing the right statistical replacement based on column data type

1. Mean Imputation

df["Salary"].fillna(df["Salary"].mean())

Best for: Normally distributed numerical data without extreme outliers.

2. Median Imputation

df["Salary"].fillna(df["Salary"].median())

Best for: Skewed numerical data or datasets with high outlier values.

3. Mode Imputation

df["Dept"].fillna(df["Dept"].mode()[0])

Best for: Categorical columns where the most frequent category is expected.

4. Constant Label

df["Dept"].fillna("Unknown")

Best for: Categorical data where missingness should be explicitly tracked.

10. 🔥 Mean vs. Median: The Outlier Trap

Why blindly using mean can corrupt your analysis

Consider 5 employees with the following salaries:$45,000, $50,000, $55,000, $60,000, and $500,000 (CEO).

Mean: $142,000

The CEO's $500k salary pulled the average up to $142,000. If an intern had a missing salary and you imputed it with the mean ($142k), you severely distorted your data!

Median: $55,000

The median sorts the values and takes the true center ($55,000). It is immune to the CEO's extreme outlier, making it far safer for real-world imputation!

11. 🔥 Live fillna() Practice

Apply appropriate strategies across numeric and categorical columns

fillna_lab.py

# Dataset has missing Age, Salary, and Department
# Task 1: Fill missing Age with median


# Task 2: Fill missing Salary with median


# Task 3: Fill missing Department with "Unknown"


# Display the cleaned DataFrame

Click "Run Python" to test fillna...

12. 🔥 Dropna vs. Fillna: The Decision Matrix

How professional analysts decide which approach to apply

When to use dropna()

Missing rate is tiny (< 1-2% of total rows)
Missing value in primary key / target variable
Imputation would falsify legal or financial records

When to use fillna()

Missing rate is moderate (5% to 30%)
Dropping would destroy sample size and bias metrics
Reasonable proxy exists (median, mode, "Unknown")

13. 🔥 Practical Data Analytics Scenario

Diagnosing an enterprise employee sales dataset

You receive the following raw quarterly dataset:

Employee	Department	Age	Salary	Performance
Amit	Sales	24	45,000	82
Priya	HR	NaN	50,000	76
Rahul	NaN	27	NaN	91
Neha	IT	25	65,000	NaN
Karan	Sales	NaN	55,000	88
Riya	NaN	29	NaN	94

Analyst Diagnostic Assessment:

What happens if we run df.dropna()? Only 1 row (Amit) remains! 83% of the dataset would be destroyed!
Department: 2 missing values. Impute with "Unknown" to preserve departmental reporting.
Age: Missing values are Priya (HR) and Karan (Sales). Impute with median (26).
Salary: Missing values are Rahul and Riya. Impute with median ($52,500).
Performance: Missing value is Neha. Impute with mean (86.2) or leave as unrated.

14. The 6-Step Professional Cleaning Workflow

A reliable framework used by senior data analysts

Inspect

df.head(), df.info()

Identify

df.isna().sum()

Diagnose

Calculate % missing

Strategy

Drop vs Impute

Clean

dropna() / fillna()

Verify

df.isna().sum() == 0

Step 6 is Essential: Always run df.isna().sum() immediately after cleaning to verify that all null counts have dropped to zero!

15. 🔥 Final Live Challenge: Clean & Verify Enterprise Data

Execute the full end-to-end cleaning pipeline

Clean the enterprise workforce dataset below and confirm that df.isna().sum() reports 0 missing values across all columns:

final_challenge.py

# Final Enterprise Cleaning Challenge
# 1. Fill missing Age with median


# 2. Fill missing Salary with median


# 3. Fill missing Department with "Unknown"


# 4. Fill missing Performance score with mean


# 5. Verify that no nulls remain!

Click "Run Python" to test your pipeline...

16. Common Mistakes to Avoid

Pitfalls made by junior analysts when handling missing data

1. Confusing missing values with zero

A zero is an actual recorded value (such as 0 defects or 0 balance). Missing values mean no data was gathered.

2. Blindly running dropna() without measuring loss

If missingness is scattered across 10 columns, dropping every row with a null can wipe out 50%+ of your training or reporting data.

3. Filling every numerical null with 0

Filling Age or Salary with 0 introduces massive distortion. Use median or mean instead.

4. Forgetting to verify after cleaning

Never assume your code worked without checking. Always conclude with df.isna().sum() to prove zero nulls remain.

17. Knowledge Check Quiz

Test your understanding of missing value detection and remediation

TEST YOUR KNOWLEDGE

Python Missing Values Knowledge Check

Validate your understanding of isna(), notna(), dropna(), and fillna() strategies before moving to the next module.

Question 1 of 4Current Score: 0 / 0

How is a missing value (NaN / None) fundamentally different from a numeric 0 or string "Unknown"?

18. Key Takeaways Checklist

What you can now do with Python Data Cleaning

Differentiate missing values (NaN/None) from zeros and blanks.

Detect missing values with isna() and isna().sum().

Measure missingness percentages to gauge the severity of data loss.

Evaluate when to drop rows vs columns with dropna().

Apply robust median imputation when outliers are present.

Follow the 6-step cleaning workflow and verify cleaned datasets.