Actor-Critic architecture diagram
Phase 5 · Reinforcement Learning Algorithm #20 Policy + Value

Actor-Critic

Two learners, one loop — a policy that acts and a value function that judges.

Core Concept
Actor-Critic fuses the two great families of reinforcement learning. The Actor is a policy that decides which action to take; the Critic is a value function that estimates how good the current situation is. The Critic evaluates each move and hands the Actor a low-variance learning signal — the advantage — telling it whether the action was better or worse than expected. This pairing keeps the sample-efficiency of value methods and the flexibility of policy methods.
Key Components

🎭 Actor (Policy Network)

Maps states to a distribution over actions. Updated in the direction the Critic says is advantageous.

⚖️ Critic (Value Network)

Estimates V(s) — expected return from a state. Learns by minimizing TD-error.

📈 Advantage Function

A(s,a) = Q(s,a) − V(s). How much better an action was than the baseline. Guides Actor updates.

🎯 Variance Reduction

The value baseline shrinks the noisy gradient of pure policy methods, stabilizing learning.

How It Works
  1. Actor observes state s and samples an action a from its policy π(a|s).
  2. Environment returns reward r and next state s′.
  3. Critic computes the TD-error / advantage from what actually happened vs. its prediction.
  4. Critic updates V(s) to reduce that error; Actor nudges π toward actions with positive advantage.
  5. Loop repeats — the two networks co-adapt until the policy is near-optimal.
A(s,a) = r + γ·V(s′) − V(s)  →  ∇θ log π(a|s) · A(s,a)
Real-World Applications
🤖 Robotic locomotion & control 🎮 Game AI (A2C / A3C) 🚗 Autonomous driving policies 💬 RLHF for language models 📦 Resource scheduling
Checkpoint Questions
  1. What is the specific job of the Critic, and how does its signal help the Actor improve?
  2. Why does subtracting a value baseline (using the advantage) reduce variance compared to pure policy gradient?
  3. Write the advantage estimate using the one-step TD-error. What does each term represent?