BERT
Bidirectional Encoder Representations from Transformers
Core Concept
BERT takes the Transformer and throws away the decoder. What remains is a deep encoder stack that reads an entire sentence in both directions at once — so the vector for a word is shaped by everything to its left and everything to its right.
The breakthrough wasn't the architecture; it was the training recipe. Hide 15% of the words and make the model guess them. That single trick turns any pile of raw text into free supervision. Pre-train once on 3.3 billion words, then bolt a tiny task head on top and fine-tune for a couple of epochs — and you beat models that were hand-built for that task.
Key Components
Encoder-only stack
12 blocks (base) or 24 (large). Each = multi-head self-attention → feed-forward. No causal mask, so every token sees every other token.
Masked Language Modeling
Randomly mask 15% of tokens; predict them via softmax over the 30k vocabulary. This is what forces bidirectionality to be useful.
Next Sentence Prediction
Given sentences A and B, does B actually follow A? A binary label read off the
[CLS] vector. Teaches inter-sentence relationships.
Special tokens & embeddings
[CLS] = sentence summary slot, [SEP] = separator.
Every input = token + segment + position embedding, summed.
How It Works
-
Tokenize and embed
WordPiece splits text into subwords. Build the sequence
[CLS] the cat sat on the [MASK] [SEP]. Each position's vector is the sum of three learned embeddings: token identity, segment (A or B), and absolute position. -
Push through the encoder stack
Each block lets every token attend to every other token. No mask means no direction bias — this is the bidirectional part.
Attention(Q,K,V) = softmax( Q Kᵀ / √d_k ) V -
Pre-train on two self-supervised objectives
MLM loss on the masked positions + NSP loss on the
[CLS]vector, optimized jointly over BooksCorpus + Wikipedia. Days on TPUs — done once, by someone else, and you download the weights. -
Fine-tune with a swapped head
Keep the trunk. Add one small output layer for your task:
[CLS]→ softmax for classification, per-token → labels for NER, start/end logits for QA. Train 2–3 epochs on your (small) labelled set. -
Why this was a phase change
Before BERT: one bespoke model per NLP task. After BERT: one pre-trained trunk, many cheap heads. This is the transfer-learning moment for language — the direct ancestor of every foundation model you use today.
Real-World Applications
- Search Google Search has used BERT to interpret query intent since 2019 — especially the prepositions and small words that keyword matching throws away.
- Support Intent classification and ticket routing: fine-tune on a few thousand labelled tickets and outperform years of hand-written rules.
- Extraction Named-entity recognition over contracts, clinical notes, and filings (BioBERT, SciBERT, FinBERT are domain-specific descendants).
- QA Extractive question answering — pick the answer span inside a passage (SQuAD-style), the backbone of early retrieval-augmented systems.
- Retrieval Sentence-BERT embeddings for semantic similarity, deduplication, and the vector search that powers modern RAG pipelines.
Checkpoint — answer before you move on
- Why does BERT need masking to be bidirectional at all? Think about what happens if a bidirectional model is asked to predict the next word — what does it see?
- What is the
[CLS]token actually for, and why can a single vector represent a whole sentence? Trace what attention does to that position across 12 layers. - BERT cannot generate text. Given its architecture and training objective, explain precisely why — and what you'd change to make it a generator. Two answers here: one about the attention mask, one about the objective.
gpt-image-2 hit its billing hard limit this run — hero authored as vector and rasterized on-VM, so the streak holds.