Sampling: how a model actually picks the next word
temperature sets the odds. sampling is the hand that reaches in and draws.
The model hands you a cloud of odds; sampling is the hand that reaches in and closes around exactly one word.
Ask a model for the next token and it does not answer with a word. It answers with a weather report over its entire vocabulary: a probability for every one of fifty-thousand-odd tokens, almost all of them vanishingly small, a handful carrying real mass. Temperature has already decided how peaked or flat that landscape is. But a probability cloud is not a sentence. Someone has to step in and choose. That step is sampling, and it is where a model's personality is actually set.
The two extremes, and why neither works
The simplest rule is greedy decoding: always take the single highest-probability token. It is fluent and it is dull, and worse, it gets stuck: feed it back its own output and it will happily write "the best the best the best" until you stop it. The opposite rule is pure sampling: treat the full distribution as an honest die and roll it. Now you get variety, but you also hand the model a one-in-ten-thousand chance, every single token, to derail into a word that makes no sense. Fluency and surprise are pulling in opposite directions.
The fix is to sample, but not from everything. Throw away the long tail of implausible tokens first, renormalize what is left so it still sums to one, then roll the die only over the survivors. The question becomes: where exactly do you cut the tail? That single decision is what separates the methods, and it is what the panel below lets you feel directly.
Top-k asks "how many candidates?" Top-p asks "how much confidence?", and only the second question knows when the model is unsure.
Why nucleus beats a fixed k
Top-k keeps a fixed count, say the 40 most likely tokens, then samples among them. The trouble is that 40 is the wrong number twice. When the model is confident, almost all of its mass sits on the top two or three tokens; the other 37 are near-zero junk that you have just invited back into the room. When the model is genuinely uncertain, 40 may be too few, chopping off plausible continuations a writer would actually consider. A fixed k cannot tell the difference between sure and unsure, because it never looks at the probabilities themselves, only at their rank.
Top-p, or nucleus sampling, fixes the mass instead of the count. Keep the smallest set of tokens whose probabilities add up to p (0.9, say) and discard the rest. Now the pool sizes itself. After "the capital of France is" the model is certain, the nucleus collapses to a single token, and you get "Paris." After "my favorite color is" the model is wide open, the nucleus swells to a dozen reasonable words, and you get genuine variety. Same threshold, completely different candidate sets, because the cutoff is reading the model's confidence in real time. That is why p, not k, is the knob most people reach for, and why the two are often used together: a generous top-p to stay adaptive, a top-k cap as a hard safety rail against the absolute worst of the tail.