How LLMs Pick Their Next Token: Logits, Temperature and Top-p

When a language model generates text, it faces a fundamental task: choosing the next word from thousands of candidates. That choice is not random; it is governed by probabilistic mechanisms. Understanding logits, temperature and top-p is the key to controlling LLM behavior.

What are logits and softmax

For every possible token, the network computes a numerical score called a logit. It is the raw signal from the network's last layer. The word "creativity" might get a logit of 5.2, while "telephone" gets 2.1. The softmax function turns these numbers into probabilities (from 0 to 1) that sum to 1.

Imagine the model is processing "machine learning is". It computes logits for every word in its vocabulary. After softmax, the distribution looks like this: "science" = 35%, "art" = 8%, "penguin" = 0.001%. The model samples randomly from this distribution: it picks likely options more often and unlikely ones rarely.

Temperature: the behavior dial

Temperature is a multiplier applied to the logits before softmax. The math is simple, but the effect is strong:

• T < 1 (for example, 0.3) pushes the model to the "icy" side. The distribution sharpens, and unlikely options get negligible probabilities. The model chooses almost deterministically, and answers are predictable and precise. Ideal for code, facts and instructions.
• T = 1 is the standard behavior: logits are used as they are.
• T > 1 (for example, 1.5–2.0) pushes the model to the "fiery" side. The distribution flattens, and unlikely options get a chance. The model chooses more randomly. Ideal for creative writing, but it risks producing errors or hallucinations.

High temperature makes the model adventurous; low temperature makes it conservative.

Top-p: smart sampling

Top-p (nucleus sampling) addresses a problem: how do you keep the model from saying complete nonsense while leaving it some freedom? The algorithm sorts tokens by probability and takes top tokens until their cumulative probability reaches p (usually 0.9).

Example: if top-p = 0.9 and the distribution is:

• "science" = 40%
• "path" = 30%
• "knowledge" = 15%
• "freedom" = 10%
• "pen guin" = 5%

The model takes the first four options (40+30+15+10=95%) and discards "penguin". Top-p is dynamic: in one context it may keep 5 options, in another 200.

What this means

These three parameters are not magic at all; they are control tools. A developer picks temperature and top-p based on the task: code generation calls for low temperature (reliability matters more than creativity), while writing a story calls for high temperature (variety). Understanding these mechanisms turns working with an LLM from a black box into an engineering task.