Algorithm 52 · Phase 10

Seq2Seq

Encoder–Decoder networks: turning one sequence into another, when neither length is known in advance.

🧠 Encoder–Decoder 🎨 technical-poster ⏱ 12 min read
Seq2Seq encoder-decoder architecture infographic
Encoder → fixed-size context vector → decoder, unrolled through time

Core Concept

Classifiers map an input to one label. Seq2Seq maps an input sequence of length T to an output sequence of length T′ — and T′ need not equal T. An encoder RNN/LSTM reads the whole input and squeezes it into a single fixed-size context vector; a decoder RNN then unrolls that vector into output tokens, one at a time, each step conditioned on what it already emitted.

The 2014 insight (Sutskever et al., Cho et al.) was decoupling reading from writing. Before this, sequence models had to emit one output per input step — so translation, summarisation, and dialogue were structurally out of reach.

Key Components

01

Encoder

Consumes x₁…x_T, updating a hidden state at each step. Its outputs are discarded — only the final state matters. Often stacked and bidirectional.

02

Context Vector

c = h_T. One fixed-size vector carrying the entire input's meaning. This is the architecture's power and its bottleneck.

03

Decoder

Initialised from c, starts at <SOS>, and generates autoregressively until it emits <EOS> — which is how it chooses its own length.

04

Teacher Forcing

During training feed the true previous token, not the model's guess. Trains fast, but creates exposure bias at inference.

How It Works

  1. Embed and read. Tokenise the source, embed each token, and step the encoder forward: hₜ = f(hₜ₋₁, xₜ) for t = 1…T.
  2. Compress. Take the final hidden (and cell) state as the context vector c. The variable-length input is now a fixed-size summary.
  3. Seed the decoder. Initialise the decoder state with c and feed it the <SOS> token.
  4. Generate autoregressively. At each step, project the decoder state through a softmax over the vocabulary, pick a token, and feed it back in as the next input.
  5. Stop on your own terms. Halt when <EOS> is emitted or a max-length cap is hit. Output length is decided by the model, not the input.
  6. Train end-to-end. Cross-entropy on every output position, backpropagated through both networks — with teacher forcing supplying the previous token.
  7. Decode better at inference. Greedy decoding takes the argmax each step and can never recover from an early mistake; beam search (algorithm 53, tomorrow) keeps k partial hypotheses alive instead.
What the model actually factorises P(y₁…y_T′ | x₁…x_T) = ∏t=1..T′ P(yₜ | y<t, c)

Real-World Applications

🌍
Machine translationThe original target task — Google Translate moved to neural Seq2Seq in 2016, replacing phrase-based statistical MT.
📝
Abstractive summarisationRead a long article, write a short novel sentence — not extract one. Output length is genuinely independent of input length.
🗣
Speech recognition & TTSAudio frames → characters (Listen-Attend-Spell), or text → mel spectrograms (Tacotron).
💬
Dialogue and codeNeural conversational models, grammatical error correction, and NL→SQL all inherit this encoder–decoder skeleton.
⚡
Ancestor of the TransformerSame encoder–decoder contract, RNN swapped for self-attention. Understanding Seq2Seq is how the Transformer stops looking arbitrary.

The Mistake Beginners Make

Treating the context vector as free capacity

A single 512-d vector must hold a 40-word sentence as faithfully as a 4-word one. Performance degrades sharply as input length grows — Cho et al. measured exactly this. Attention was invented as the fix: let the decoder look back at all encoder states instead of one summary. So if your Seq2Seq model is fine on short inputs and collapses on long ones, that is not a bug in your training loop — it is the bottleneck doing what bottlenecks do.

Checkpoint — answer before you move on

  1. Why can Seq2Seq produce an output sequence of a different length than its input, when a plain RNN tagger cannot? Hint: what decides when generation stops?
  2. What exactly is the information bottleneck in vanilla Seq2Seq, and what empirical symptom does it produce? Hint: think about what changes as sentence length grows.
  3. Teacher forcing speeds up training but introduces exposure bias. Describe the mismatch between training and inference that causes it. Hint: what is fed as the previous token in each regime?