Python EDA: Correlation & Linear Relationships
Master bivariate relationship analysis with Python and Pandas. Learn how the Pearson correlation coefficient r quantifies linear direction and strength, experiment with live interactive scatter plots, calculate pairwise metrics and correlation matrices, discover how outliers and non-linear patterns distort findings, and develop the discipline to separate association from causality.
1. What is Correlation?
Measuring whether two numerical variables tend to move together
In data analytics, correlation measures the statistical association between two continuous numerical variables:
Positive Association
As variable A increases, variable B tends to increase.
• Example: Advertising Spend ↑ and Website Visits ↑.
Negative Association
As variable A increases, variable B tends to decrease.
• Example: Product Price ↑ and Unit Demand ↓.
2. The Pearson Correlation Coefficient (r)
A normalized scale strictly bounded between -1.0 and +1.0
Points align on a straight line sloping downwards.
Points scatter randomly without a linear slope.
Points align on a straight line sloping upwards.
1. Direction (Sign)
Indicated by the + or - sign. Positive means both move upward together; negative means opposite directions.
2. Strength (Magnitude)
Indicated by absolute value |r|. An r = -0.85 is just as strong as r = +0.85!
3. 🔥 Live Interactive Scatter Plot Lab
Click anywhere on the canvas to add/remove points and watch Pearson r recalculate in real time
4. Calculating Correlation in Pandas: Series.corr()
Computing Pearson r between paired numerical columns
In Pandas, use Series.corr() to compute the pairwise Pearson correlation:
5. Negative Correlation: Price vs. Demand
Quantifying inverse economic associations
When evaluating pricing elasticity, higher prices often coincide with lower unit demand:
6. The Golden Rule: Correlation ≠ Causation
Co-movement never proves a direct cause-and-effect mechanism
During summer, ice cream sales soar (r = +0.85 with swimming pool drownings). Does eating ice cream cause swimmers to drown? Of course not!
Both variables are independently driven by an unobserved third factor: Hot Summer Temperatures!
Statistical Correlation
Two numerical sequences move up or down in sync within a sample. Verified by calculating Pearson r.
Scientific Causation
Intervening on X directly produces a change in Y. Requires controlled randomized A/B testing, domain science, and temporal isolation.
7. Critical Limitations: Outliers & Non-Linear Curves
Why analysts must ALWAYS visualize with scatter plots before trusting r
1. Outlier Distortion
Because Pearson r relies on mean and standard deviation, a single corrupted extreme point can pull a +0.95 correlation down to +0.20 (or create a false correlation where none exists).
2. Non-Linear Relationships
Pearson correlation only measures straight lines. A perfect quadratic U-curve (e.g. employee stress vs performance) has r ≈ 0.00 even though the relationship is 100% predictable!
8. Full Correlation Matrix: df.corr(numeric_only=True)
Pairwise cross-comparison across all numerical metrics in a table
Running df.corr(numeric_only=True) computes an N×N matrix where each cell is the correlation between column pairs:
| Metric | Advertising | Sales | Price | Units |
|---|---|---|---|---|
| Advertising | 1.0000 | +0.9965 | -0.9782 | +0.9984 |
| Sales | +0.9965 | 1.0000 | -0.9715 | +0.9958 |
| Price | -0.9782 | -0.9715 | 1.0000 | -0.9842 |
| Units | +0.9984 | +0.9958 | -0.9842 | 1.0000 |
9. 🔥 Final Live Challenge: Student Exam Performance
Analyze StudyHours vs ExamScore and SleepHours to uncover bivariate dynamics
You are given an 8-student dataset with StudyHours, ExamScore, and SleepHours. Compute the pairwise correlations, generate the correlation matrix, and critically evaluate the sleep-study tradeoff:
10. Common Mistakes to Avoid
Pitfalls made when evaluating bivariate relationships
Just because A and B correlate does not mean A caused B. A confounding third factor (or reverse causality) frequently explains the link.
A correlation of r = -0.92 is not "weaker" than +0.50. The minus sign only denotes inverse direction; 0.92 is far stronger.
Anscombe’s Quartet famously proved that identical correlation numbers can represent completely different scatter patterns (including curved arcs and extreme outliers).
Pearson r only checks for straight linear lines. Symmetrical curves (like parabolas or sine waves) produce r = 0 despite having an exact mathematical relationship.
11. Knowledge Check Quiz
Validate your correlation and EDA mastery
Python Correlation Knowledge Check
Validate your understanding of Pearson r, direction vs strength, scatter plots, correlation matrices, and causation pitfalls.
What does a Pearson correlation coefficient of r = -0.89 between product price and demand signify?
12. Key Takeaways Checklist
Skills you have mastered in Correlation Analysis
Series.corr().df.corr(numeric_only=True).