Actor-Critic Methods
Core Concept
Pure policy gradient (REINFORCE) is unbiased but wildly noisy — it waits for a whole episode's return before it learns anything, and that return swings hard. Pure value methods (Q-learning) are stable but struggle with continuous or high-dimensional actions. Actor-Critic splits the job: the actor is a policy π(a|s;θ) that picks actions; the critic is a value function V(s;w) that scores the state it landed in. The critic's prediction error becomes the actor's learning signal — so the actor can update every step, not every episode.
Read δ as a surprise meter. Positive: that action turned out better than the critic expected — make it more likely. Negative: worse than expected — make it less likely. Zero: the critic already saw it coming, so there is nothing to learn.
Key Components
Policy π(a|s;θ)
Outputs an action distribution (softmax for discrete, Gaussian mean/σ for continuous). Trained by policy gradient, weighted by the critic's signal. This is the thing you actually deploy.
Value V(s;w)
Estimates expected return from a state. Trained by TD regression on r + γV(s′). Never chooses actions — it exists purely to be a baseline that cancels noise.
Advantage A(s,a)
A(s,a) = Q(s,a) − V(s): "how much better than average was this action here?" Subtracting V leaves the gradient unbiased but slashes variance. δt is its cheapest estimator.
GAE(γ, λ)
Generalized Advantage Estimation blends n-step errors: Â = Σ(γλ)kδt+k. λ→0 gives low-variance/high-bias TD; λ→1 gives Monte-Carlo. λ≈0.95 is the workhorse default.
How It Works
- Act. Observe st, sample at ~ π(·|st;θ), step the environment, receive rt and st+1.
- Criticize. Compute the TD error δt = rt + γV(st+1) − V(st). Bootstrap with V(st+1)=0 if the episode terminated (a classic bug source).
- Update the actor. Ascend δt·∇log π. Treat δt as a constant here — stop the gradient through the critic, or the actor learns to game the value head instead of the environment.
- Update the critic. Descend the squared TD error, ½δt2, so future baselines are sharper.
- Regularize. Add an entropy bonus +c·H(π) to the actor loss. Without it, the policy collapses to one action early and never explores again.
- Parallelize. A2C runs N synchronized environments and averages the batch; A3C runs N asynchronous workers that each push gradients to a shared global net. Both decorrelate samples — which is what replay buffers do for DQN.
Variance vs Bias
| Method | Signal | Variance | Bias |
|---|---|---|---|
| REINFORCE | full return Gt | Very high | None |
| + baseline | Gt − V(s) | High | None |
| Actor-Critic (TD) | δt (1-step) | Low | Some |
| GAE(λ) | blended n-step | Tunable | Tunable |
This table is the whole design space. Every modern policy-gradient algorithm — A2C, PPO, SAC, TRPO — is a different answer to the same question: how much bias will I accept to stop the gradient from screaming?
Real-World Applications
Checkpoint — answer before you move on
- Why does subtracting the baseline V(s) from the return reduce variance without introducing bias into the policy gradient? Hint: what is the expected value of ∇log π(a|s) under π itself?
- The TD error δt is a biased estimate of the true advantage, yet Actor-Critic usually learns faster than unbiased REINFORCE. Explain the trade being made. Hint: think about total error = bias² + variance, and about updating per-step vs per-episode.
- What does the λ in GAE(γ, λ) actually control, and what would you expect to see in training if you set λ = 1 versus λ = 0? Hint: λ=1 recovers Monte-Carlo returns; λ=0 recovers the 1-step TD error.