The Difference Between Telling and Showing a Model What You Want
Subtitle: Instructions name the task. Examples reveal its shape.
- Panel 1 - Zero-shot: the model must guess:
- The prompt gives an instruction only
- No examples. No output template.
- Pretraining supplies the likely shape
- Works well for familiar task types
- Silent failure: right idea, wrong shape
- Panel 2 - Few-shot: examples become the spec:
- Put input-output pairs in the prompt
- Their shared pattern narrows the answer
- The model infers the task in context
- Weights never change during this step
- This is steering, not new training
- Panel 3 - The trade, measured:
- SuperGLUE zero-shot: 52.0
- One-shot: 55.4, a gain of 3.4
- Thirty-two-shot: 71.8, up 19.8
- Examples cost input tokens and time
- More examples face diminishing returns
- Panel 4 - Falsify 'just add more':
- Add examples only after a baseline
- Target the failure with 2-5 examples
- Sweep the count; plot quality vs tokens
- Shuffle order: scores can move 10+ points
- Stop if quality plateaus or falls
- Crowded context can squeeze out the query