Greedy vs Random: How the Next Token Is Chosen From a Probability Distribution

• THE MODEL OUTPUTS A GUESS, NOT A WORD. Final linear layer + softmax -> ~50,000 probabilities summing to 1 over the whole vocabulary. Not text: one guess spread over every possible word. The context 'coded' into a probability distribution (Shannon analogy).
• DECODING IS THE SEPARATE LOOP. Decoding = turn that distribution into one token, feed it back, repeat. Transformer ENCODES input into a distribution; the decoder turns distributions into a sequence, one token at a time. Training made the distributions good; decoding only consumes them at inference.
• TWO FAMILIES OF STRATEGY. DETERMINISTIC: greedy = argmax, always top-1. Fast, no randomness. Locks onto the top-probability attractor -> repetitive generic text. 'Safe, but boring.' STOCHASTIC: sampling = draw a token from the distribution. Temperature T rescales logits before softmax: T<1 sharpens, T>1 flattens. top-k keeps the k highest. top-p (nucleus) keeps the smallest set summing to p. Exploration comes from here.
• FALSIFICATION: GREEDY IS A LIMIT CASE. Greedy is NOT 'better' or 'simpler' math. It is sampling with T -> 0, the distribution collapsed to a point. Test: greedy output must equal T=0 sampling exactly on the same prompt. If greedy ever produced genuinely varied prose, the attractor story is wrong.