Policy Gradient (REINFORCE) sketchnote
Phase 5 · Reinforcement Learning · Algorithm 19 / 30

Policy Gradient — REINFORCE

Learn the policy directly, not the value of states.

🎯 Daily Micro-Learning Card

📜 Core Concept

Instead of learning how good each state/action is (value-based methods like Q-learning) and then acting greedily, policy gradient methods parameterize the policy itself — a network that outputs a probability distribution over actions — and nudge those probabilities up for actions that led to high return and down for actions that led to low return. You climb the expected-reward hill by gradient ascent.
∇θ J ≈ Σt ∇θ log πθ(at|st) · (Gt − b)
reinforce action probability, weighted by return minus a baseline
Mental model: “Try, then turn up the volume on what worked.” Play a full episode, see the total reward, and make the good moves more likely next time — in proportion to how good the outcome was.

🔑 Key Components

Parameterized Policy πθ

A network mapping state → action probabilities. Differentiable, so we can push gradients through it.

Expected Return J(θ)

The objective we maximize: average total reward the policy earns over episodes.

Monte Carlo Rollout

Sample a full trajectory, compute the actual return Gt — no bootstrapping, unbiased but noisy.

Baseline b

Subtract a reference (e.g. average return) to cut variance without adding bias.

⚙️ How It Works (REINFORCE)

Run the current policy πθ for a full episode, recording states, actions, and rewards.
Compute the return Gt = discounted sum of rewards from each timestep to the end.
Subtract a baseline b to center the returns and shrink gradient variance.
Update: θ ← θ + α Σ ∇ log π(a|s)(Gt−b) — gradient ascent toward more reward.
Repeat across many episodes; probabilities of high-return actions steadily rise.

🌎 Real-World Applications

Robotics control Continuous action spaces Game-playing agents LLM alignment (RLHF/PPO roots) Resource scheduling Portfolio/trading policies

🧪 Checkpoint Questions

1. Why optimize the policy directly instead of learning Q-values first?
Hint: think about continuous or high-dimensional action spaces where argmax over Q is intractable, and about learning stochastic policies.
2. How does subtracting a baseline reduce variance without introducing bias?
Hint: the expected value of ∇ log π times a constant baseline is zero — it re-centers the reward signal but leaves the gradient's mean unchanged.
3. Why is vanilla REINFORCE high-variance, and what makes it slow to converge?
Hint: it uses full Monte Carlo returns from noisy single trajectories — one lucky/unlucky episode swings the whole update. This motivates Actor-Critic (next up).