Phase 2 · Ensembles
Algorithm 9 / 30
Boosting Family
AdaBoost
Adaptive Boosting — turning many weak learners into one strong classifier
💡Core Concept
AdaBoost builds a strong classifier by training a sequence of weak learners
(usually one-split decision stumps), where each new learner is forced to pay
more attention to the examples the previous ones got wrong. It does this by
re-weighting the training samples every round. The final prediction is a
weighted majority vote — accurate learners get louder votes. It was the first
practical boosting algorithm and remains a beautiful lesson in the power of the crowd.
🧩Key Components
Weak Learners
Simple models (decision stumps) that only need to be slightly better than random guessing.
Sample Weights
Every training point carries a weight; misclassified points get boosted so the next learner focuses on them.
Learner Weight α
Each stump earns a vote α based on its error — lower error means a stronger voice in the final vote.
Weighted Vote
Final output is the sign of the α-weighted sum of all weak learners' predictions.
⚙️How It Works
- Initialize all sample weights equally (1/N).
- Train a weak learner on the weighted data; measure its weighted error ε.
- Compute its vote α = ½·ln((1−ε)/ε) — smaller error means larger α.
- Increase weights on misclassified samples, decrease on correct ones, then renormalize.
- Repeat for T rounds, then combine via weighted majority vote.
H(x) = sign( Σₜ αₜ · hₜ(x) ) | αₜ = ½ ln((1−εₜ)/εₜ)
🌍Real-World Applications
📷
Face detection — the Viola-Jones detector powering early digital-camera face boxes.
💳
Fraud & churn scoring — tabular classification where interpretable stumps help.
🩺
Medical diagnosis — combining weak indicators into a reliable risk signal.
🔠
Text / spam classification — boosting simple word-presence rules.
🎯Checkpoint Questions
1. Why does AdaBoost increase the weight of misclassified samples after each round — what behavior does this force in the next learner?
2. In α = ½·ln((1−ε)/ε), what happens to a learner's vote as its error ε approaches 0.5? What does that tell you about a stump that's no better than guessing?
3. AdaBoost reduces bias by adding learners sequentially. How does this differ from how Random Forest (bagging) reduces variance in parallel?