Temperature is not a creativity knob
It does one tiny thing: it divides the model's scores before they become probabilities, and that reshapes everything.
The slider you think controls creativity is really a single division performed on a column of numbers, a heartbeat before the model speaks.
Open any model playground and you will find a slider called temperature, usually running from 0 to 2, often with a tooltip whispering the word creativity. Drag it up and the prose gets stranger; drag it down and the model turns stiff and repetitive. From this almost everyone concludes the same thing: temperature is the randomness knob, the imagination dial. It is a tidy story, and it is wrong in a way that hides what the model is actually doing.
What the model actually produces
A language model does not choose a word. At each step it produces a long list of raw scores: one number, called a logit, for every token in its vocabulary. These logits are not probabilities; they are just unbounded scores, a kind of confidence in arbitrary units. To turn them into something you can sample from, the model runs them through softmax, which exponentiates each score and normalizes so they sum to one. Out comes a probability for every possible next token. Temperature is the one operation that sits in between: before softmax sees the logits, every logit is divided by a single number, T.
That division is the whole trick. Because softmax exponentiates, dividing the logits by a small T (say 0.2) stretches the gaps between scores apart: the leading token's probability rockets toward one, and everything else collapses toward zero. Divide by a large T (say 1.8) and you compress those gaps: the front-runner loses its commanding lead, and tokens that were near-impossible suddenly hold real probability mass. Same logits, different sharpness. Move the slider below and watch the bars breathe.
The model's knowledge is fixed. Temperature does not add ideas; it only decides how willing the model is to bet on its second-best guess.
The two ends, and the tail
At T→0 the distribution becomes a spike: the single highest-scoring token wins every time. This is greedy decoding, and it is essentially deterministic: feed the same prompt and you get the same answer, word for word. That is why low temperature is right for arithmetic, code, and structured extraction, where there is one correct continuation and confident commitment is a feature. Push T up and the curve flattens; the model starts reaching past its favorite into plausible alternatives, which reads to us as range, surprise, voice. But there is a cliff. Past roughly 1.2 to 1.5, the curve flattens so far that genuinely unlikely tokens win often enough to break grammar and meaning: the text derails into fluent-looking noise.
This is also where top-p and top-k come in, and why they are not rivals to temperature but partners. Top-k keeps only the k highest-scoring tokens; top-p (nucleus sampling) keeps the smallest set whose probabilities sum to p. Both amputate the long tail of garbage tokens before sampling, so you can raise temperature for variety without inviting the worst candidates to the table. The headline, though, stays simple: temperature is not creativity and not randomness. It is a single division that reshapes how peaked one fixed set of scores becomes, and once you see it that way, you can tune for variety and accuracy at the same time, instead of treating it as a mood.