Linear Regression infographic
Phase 1 · Supervised Learning Algorithm #1 Regression

Linear Regression

The Best-Fit Line Finder — predicting continuous values by minimizing error.

What It Does
Linear Regression predicts continuous values (house prices, temperatures, sales figures) by finding the best-fit line through your data points. Mental model: imagine stretching a rubber band through scattered points — the band settles where it minimizes total distance from all points. Input is features (x), output is a continuous prediction (y), and the goal is to minimize error.
Key Concepts

📉 Cost Function (MSE)

Mean Squared Error measures the average squared distance between predictions and actual values. Squaring penalizes large errors and keeps the math smooth & differentiable.

🧮 Normal Equation

θ = (XᵀX)⁻¹Xᵀy. An exact algebraic solution — one computation, done. Slow for large datasets (matrix inversion is expensive).

⬇️ Gradient Descent

Iterative: take small steps downhill on the cost function. Scales well to large datasets, but requires tuning the learning rate α.

⚖️ Feature Scaling

Features on different scales distort the cost function. Normalize to similar ranges (mean=0, std=1) for faster, more direct convergence.

How It Works
  1. Start with a linear hypothesis hθ(x) = θ₀ + θ₁x mapping features to a predicted value.
  2. Measure fit with the cost function — the summed squared distance between predictions and actual values.
  3. Choose a solver: the normal equation for a direct solution, or gradient descent for large datasets.
  4. Scale features first so gradient descent converges directly instead of zigzagging.
  5. Iterate (or solve) until θ minimizes the cost — you now have the best-fit line.
J(θ) = (1/2m) Σ(hθ(x⁽ⁱ⁾) − y⁽ⁱ⁾)²  →  θ := θ − α∇J(θ)
When to Use It
✅ Predicting continuous values ✅ Roughly linear relationships ✅ Need interpretable coefficients ✅ Baseline before complex models ❌ Categorical output → classifier ❌ Highly non-linear → polynomial/other
Checkpoint Questions
  1. Can you derive the normal equation from the cost function? (Hint: take the derivative of J(θ), set to zero, solve for θ.)
  2. Why minimize squared error instead of absolute error? (Hint: think about differentiability and outlier sensitivity.)
  3. What happens when features aren't scaled? (Hint: visualize the cost function contours.)