Q-Learning
Learning what to do by trying, failing, and remembering what paid off.
🎯 Core Concept
Q-Learning is a model-free, off-policy reinforcement learning algorithm. It learns a
Q-value for every (state, action) pair — an estimate of the total future reward you can
expect if you take that action now and act optimally forever after. No map of the world is
needed; the agent discovers value purely through trial-and-error interaction.
Mental model: a growing cheat-sheet. Each cell says “from here, doing
this is worth about this much down the road.” Every experience nudges one cell
closer to the truth.
🔑 Key Components
State, Action, Reward
The agent observes a state s, picks an action a, and gets reward r plus a next state s'. The (s,a,r,s') tuple is the unit of learning.
The Q-Table
A lookup table of Q(s,a) values — rows are states, columns are actions. The best action in any state is simply the column with the highest value.
Bellman Update
Each step blends the old estimate with a new target built from the reward plus the best next-state value, weighted by learning rate α.
ε-Greedy
Balances exploration (try random actions to discover) against exploitation (use current best). ε usually decays over time.
⚙️ How It Works
- Initialize the Q-table (often all zeros) and set α (learning rate), γ (discount), ε (exploration).
- In state s, choose action a via ε-greedy: usually the argmax Q-value, occasionally random.
- Take the action; observe reward r and next state s'.
- Apply the Bellman update — move Q(s,a) toward the observed target.
- Set s ← s' and repeat until the episode ends; loop over many episodes until Q-values converge.
Q(s,a) ← Q(s,a) + α [ r + γ · maxa' Q(s',a') − Q(s,a) ]
🌎 Real-World Applications
- Game-playing agents & simple grid-world / maze navigation
- Robot path planning and control in discrete environments
- Traffic-signal timing and elevator dispatch optimization
- Dynamic pricing, inventory, and resource-allocation policies
🧪 Checkpoint Questions
1. What does a single Q-value actually represent — and why the “max” over next actions?
Hint: it's expected cumulative future reward assuming optimal play from s' onward; the max encodes that optimism.
2. Why do we need exploration at all? What fails if the agent always exploits its current best?
Hint: an early lucky reward can trap the agent; unvisited (s,a) pairs never get their true value revealed.
3. Q-Learning is called “off-policy.” What does that mean, and how does the update rule show it?
Hint: it learns about the greedy policy (the max) even while behaving with a different, exploratory one.