Gradient Boosting — Sensei Learning Card

Core Concept

Gradient Boosting builds an ensemble sequentially. Each new weak learner (a small decision tree) is trained to fix the residual errors left by the models before it. Rather than voting in parallel like Random Forest, boosting descends a loss surface step by step — performing gradient descent in function space — turning a crowd of shallow trees into one sharp, low-bias predictor.

Why It Works

Each tree fits the negative gradient of the loss with respect to current predictions — the direction that most reduces error. A small learning rate shrinks each tree's contribution, trading more rounds for better generalization. Regularization (tree depth, subsampling, shrinkage) keeps the sequential process from overfitting the residuals.

Checkpoint

Why does boosting reduce bias where bagging (Random Forest) mainly reduces variance?
What role does the learning rate play, and why does a smaller value usually need more trees?
In one sentence: what does each new tree actually fit?