Deep Q-Network infographic
Algorithm #18 Phase 5 · Reinforcement Learning Deep RL

Deep Q-Network (DQN)

Q-learning meets deep neural networks — learning to act from raw high-dimensional states.

🎯 Core Concept

Tabular Q-learning stores one value per (state, action) pair — impossible when states are game pixels or continuous sensors. DQN replaces the Q-table with a neural network that approximates Q(s,a), letting an agent learn optimal policies in vast state spaces. Two tricks tame the instability of mixing bootstrapping, function approximation, and off-policy learning: an experience replay buffer and a target network.
Mental model: A student (online network) practicing against a slowly-updated answer key (target network), reviewing a shuffled deck of past experiences (replay buffer) instead of only the last move — so lessons don't chase their own tail.

🔑 Key Components

1. Function Approximation

A neural net maps state → Q-value for every action. Generalizes to unseen states instead of needing a lookup entry for each one.

2. Experience Replay

Store transitions (s, a, r, s′) in a buffer; train on random minibatches. Breaks temporal correlation and reuses data efficiently.

3. Target Network

A periodically-frozen copy of the Q-net supplies the TD target. Fixing it stops the target from moving every step, which would otherwise diverge.

4. ε-Greedy Exploration

With probability ε act randomly, else greedily. ε decays over time — explore early, exploit once Q-estimates sharpen.

⚙️ How It Works

  1. Observe state s; pick action a via ε-greedy over the online Q-net's outputs.
  2. Execute a, receive reward r and next state s′; store (s, a, r, s′) in the replay buffer.
  3. Sample a random minibatch of transitions from the buffer.
  4. Compute the TD target using the frozen target network, then minimize the squared error below by gradient descent on the online network.
  5. Every N steps, copy the online weights into the target network. Decay ε. Repeat.
L(θ) = ≈ [ ( r + γ · maxa′ Q(s′, a′; θ−) − Q(s, a; θ) )2 ]

🌎 Real-World Applications

Atari & game AI — the original DQN reached human-level play on dozens of Atari games from pixels alone.
Robotics & control — discrete-action control tasks like grasping, navigation, and balancing.
Resource management — datacenter cooling, energy scheduling, network routing decisions.
Recommendation & ads — sequential decision problems framed as maximizing long-term engagement reward.

🧪 Checkpoint Questions

1. Why does DQN need experience replay rather than learning from consecutive transitions?
(Hint: consecutive samples are highly correlated — what does that do to gradient estimates and stability?)
2. What specific instability does the target network solve, and what would go wrong without it?
(Hint: the TD target depends on the same weights you're updating — think "chasing a moving target".)
3. DQN is off-policy. How does that let experience replay work at all, and why couldn't on-policy SARSA reuse an old buffer the same way?
(Hint: compare the max over next actions in the target vs. learning from the action actually taken.)
StreakDay 19
Phase 560%
Curriculum18 / 30