Loading content...
Loading content...
Master professional missing value detection, diagnosis, and remediation. Learn why missing data happens, how to measure missingness percentages, evaluate the trade-offs of dropping vs filling, and apply robust statistical imputation (mean, median, mode, constant) to build production-ready datasets.
Understanding unavailable information in enterprise records
In real-world data analytics, datasets are rarely complete. Consider this human resources log:
| Name | Age | Salary ($) |
|---|---|---|
| Amit | 24 | 45,000 |
| Priya | NaN | 50,000 |
| Rahul | 27 | NaN |
| Neha | 25 | 60,000 |
In Pandas, missing values appear as NaN (Not a Number) for numerical data or None. This indicates that information is unavailable or unrecorded.
0 or "No"!$0 means the account is empty (a legitimate number). A bank balance of NaN means we have no idea how much money the person has. Treating missing values as 0 distorts statistical analyses!Generating boolean masks and column summary counts
Pandas provides two primary methods to locate missing values:
Returns True if missing, False if present.
Returns True if valid data, False if missing.
Sums Trues per column to give exact missing counts.
df.isna().sum() so powerful?True evaluates as 1 and False evaluates as 0. Calling .sum() counts every missing value in each column instantly!Inspect nulls and generate summary statistics
Given the df below, practice detecting and counting missing values:
Relative percentages guide the cleaning strategy
A count of 50 missing values means something very different in a dataset of 100 rows (50% missing!) versus a dataset of 1,000,000 rows (0.005% missing). Analysts always calculate the percentage of missing values:
# Calculate percentage of missing values per column missing_pct = (df.isna().sum() / len(df)) * 100 print(missing_pct)
Removing incomplete rows or empty columns
The fastest way to eliminate missing values is using dropna():
Removes any row that contains at least one NaN.
df_clean = df.dropna()Removes any column that contains at least one NaN.
df_clean = df.dropna(axis=1)Never assume: Missing value = Delete row
Deleting records is a destructive operation. Consider the trade-offs:
• Only a tiny fraction (< 1–2%) of records are incomplete.
• Critical identifier is missing (e.g. missing Customer_ID in transactions).
• The record is completely unrecoverable for the specific business question.
• 15–40% of the rows would vanish.
• Sample size shrinks drastically, ruining statistical confidence.
• Introduces bias (e.g. low-income users skipping salary fields; dropping them skews average salary upward!).
Observe how much data is removed across rows vs columns
Preserving sample size through sensible imputation
Instead of deleting records, an analyst can replace missing values using fillna():
# Replace nulls in a specific column with a value
df["Department"] = df["Department"].fillna("Unknown")
# Replace nulls with the column's median
df["Age"] = df["Age"].fillna(df["Age"].median())0 is NOT automatically correct!Age with 0, you introduce 0-year-old babies into an adult workforce, ruining the average age metric!Choosing the right statistical replacement based on column data type
df["Salary"].fillna(df["Salary"].mean())Best for: Normally distributed numerical data without extreme outliers.
df["Salary"].fillna(df["Salary"].median())Best for: Skewed numerical data or datasets with high outlier values.
df["Dept"].fillna(df["Dept"].mode()[0])Best for: Categorical columns where the most frequent category is expected.
df["Dept"].fillna("Unknown")Best for: Categorical data where missingness should be explicitly tracked.
Why blindly using mean can corrupt your analysis
Consider 5 employees with the following salaries:$45,000, $50,000, $55,000, $60,000, and $500,000 (CEO).
The CEO's $500k salary pulled the average up to $142,000. If an intern had a missing salary and you imputed it with the mean ($142k), you severely distorted your data!
The median sorts the values and takes the true center ($55,000). It is immune to the CEO's extreme outlier, making it far safer for real-world imputation!
Apply appropriate strategies across numeric and categorical columns
How professional analysts decide which approach to apply
Diagnosing an enterprise employee sales dataset
You receive the following raw quarterly dataset:
| Employee | Department | Age | Salary | Performance |
|---|---|---|---|---|
| Amit | Sales | 24 | 45,000 | 82 |
| Priya | HR | NaN | 50,000 | 76 |
| Rahul | NaN | 27 | NaN | 91 |
| Neha | IT | 25 | 65,000 | NaN |
| Karan | Sales | NaN | 55,000 | 88 |
| Riya | NaN | 29 | NaN | 94 |
df.dropna()? Only 1 row (Amit) remains! 83% of the dataset would be destroyed!"Unknown" to preserve departmental reporting.A reliable framework used by senior data analysts
df.head(), df.info()
df.isna().sum()
Calculate % missing
Drop vs Impute
dropna() / fillna()
df.isna().sum() == 0
df.isna().sum() immediately after cleaning to verify that all null counts have dropped to zero!Execute the full end-to-end cleaning pipeline
Clean the enterprise workforce dataset below and confirm that df.isna().sum() reports 0 missing values across all columns:
Pitfalls made by junior analysts when handling missing data
A zero is an actual recorded value (such as 0 defects or 0 balance). Missing values mean no data was gathered.
If missingness is scattered across 10 columns, dropping every row with a null can wipe out 50%+ of your training or reporting data.
Filling Age or Salary with 0 introduces massive distortion. Use median or mean instead.
Never assume your code worked without checking. Always conclude with df.isna().sum() to prove zero nulls remain.
Test your understanding of missing value detection and remediation
Validate your understanding of isna(), notna(), dropna(), and fillna() strategies before moving to the next module.
How is a missing value (NaN / None) fundamentally different from a numeric 0 or string "Unknown"?
What you can now do with Python Data Cleaning
isna() and isna().sum().dropna().