LLM Sampling, Demystified: Temperature, Top-k, Top-p, Min-p and Repetition Penalty

Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product. A language model does not actually "write" the next token. It produces a probability distribution over the vocabulary and says, roughly: Here are 50,000 things I could say next. These are how plausible I think they are. Everything that happens after that is decoding. And this is where things get interesting: the same model, with the same prompt, can produce radically different text depending on how you sample from that distribution. In 2019, researchers studying GPT-2 generation found that decoding strategy alone could dramatically change the character of generated text, even when the underlying neural network was identical. Greedy and beam-style decoding tended toward bland repetition; unrestricted sampling could wander into low-probability nonsense; carefully truncated sampling produced text much closer to human writing. This is why parameters such as temperature, top_k, top_p, min_p, and repetition_penalty are not merely API knobs. They are different ways of answering one question: Given the model's probability distribution, which parts of that distribution are we willing to trust? This article builds from that intuition to the actual equations, implementation details, and operational consequences. 1. First: the model gives you a probability distribution, not an answer Suppose the prompt is: The cat sat on the The model might assign probabilities approximately like: mat 0.42 floor 0.20 chair 0.12 bed 0.08 table 0.05 roof 0.01 ... The model does not intrinsically say: mat = correct floor = incorrect chair = incorrect It says: mat is very plausible floor is also plausible chair is plausible ... A decoder turns that distribution into one actual token. There are two basic philosophies. Deterministic decoding Take the highest-probability token: argmax_i P(token_i | context) This is greedy decoding. Stochastic decoding Sample from the distribution: token ~ P(token | context) Now the model can occasionally choose "floor" instead of "mat". That small distinction compounds over a sequence. Suppose the model has two reasonable choices at every step, with probabilities: 0.8 / 0.2 Over 20 independent-ish decisions, the probability of taking the 0.8 choice every time is approximately: 0.8^20 ~= 0.0115 So even modest randomness creates a large space of possible generations. That is why sampling is useful for creative writing, brainstorming, synthetic data, dialogue, and many other tasks. It is also why sampling can produce garbage. The core problem is that language-model distributions have a long tail: a few tokens may be very plausible, followed by thousands of increasingly questionable ones. The history of modern sampling tricks is largely the history of figuring out where that tail should be cut. 2. Temperature: change how sharp the distribution is Temperature is the simplest way to alter the distribution. Start with logits: z_i Before softmax, temperature modifies them as: z_i' = z_i / T Then: P_i = exp(z_i') / sum_j exp(z_j') The intuition is more important than the equation: T < 1 -> sharpen distribution T = 1 -> leave distribution alone T > 1 -> flatten distribution Imagine: Token A 0.70 Token B 0.20 Token C 0.07 Token D 0.03 Lower the temperature and the model becomes more committed to A. Raise it and probability shifts toward B, C, and D. A useful way to understand temperature mathematically is through odds ratios. Suppose: P(A) / P(B) = 4 After temperature scaling, the ratio becomes approximately: (P(A) / P(B))^(1/T) At T = 2: 4^(1/2) = 2 So a 4:1 preference becomes 2:1. At T = 0.5: 4^(1/0.5) = 16 The same underlying preference becomes 16:1. That is what temperature really does: It changes how strongly the decoder believes the model's ranking. This also explains why temperature does not solve the long-tail problem. Suppose the distribution is: A 0.40 B 0.25 C 0.15 D 0.08 E 0.05 F 0.03 G 0.02 ... Turning the temperature up does not distinguish between "reasonable alternatives" and "nonsense in the tail." It simply gives more probability to everything. Turning it down does the opposite: it pushes the model toward its favorite candidates. This is exactly the tradeoff that became visible in early large-scale text generation experiments. The GPT-2-era generation recipes commonly used combinations such as temperature 0.7 and top-k=40; Holtzman and colleagues later showed that temperature alone could not adequately control the quality/diversity tradeoff. So temperature answers: How adventurous should I be? But it does not answer: Which candidates should I consider at all? That is where top-k and top-p enter. 3. Top-k: keep exactly K candidates Top-k is almost embarrassingly simple. Sort the candidate tokens by probability and keep only the top K. For example: A 0.40 B 0.25 C 0.15 D 0.08 E 0.05 F 0.03 G 0.02 With: top_k = 3 we keep: A 0.40 B 0.25 C 0.15 Then renormalize: A 0.40 / 0.80 = 0.50 B 0.25 / 0.80 = 0.3125 C 0.15 / 0.80 = 0.1875 Then sample from those three. Top-k became a practical generation technique in work on hierarchical neural story generation by Angela Fan, Mike Lewis, and Yann Dauphin in 2018. The paper was working on a very concrete problem: getting neural models to produce long, coherent stories rather than collapsing into low-quality text. The key limitation of top-k is visible immediately. Consider two contexts. Context A: very confident model A 0.95 B 0.02 C 0.01 D 0.005 E 0.005 ... Context B: uncertain model A 0.20 B 0.18 C 0.16 D 0.14 E 0.12 F 0.10 G 0.10 ... With top_k=5, you keep five tokens in both cases. But that means very different things. In Context A, five tokens may include several absurd tail candidates. In Context B, five tokens may throw away perfectly reasonable possibilities. So the problem with top-k is: K is constant, while the model's uncertainty is not. Top-k is therefore a blunt instrument. A useful engineering intuition: top-k asks: "How many candidates may I consider?" It does not ask: "How much of the probability mass should I trust?" That second question leads directly to nucleus sampling. 4. Top-p: keep enough probability mass Top-p, or nucleus sampling, was introduced by Ari Holtzman and colleagues in their 2019 work on neural text degeneration. Their observation was important enough to change how people thought about generation: maximum likelihood is a good training objective, but simply choosing the most likely continuation is not necessarily a good generation objective. Models were producing text that could have excellent token-level likelihood while becoming bland, repetitive, or trapped in loops. Top-p changes the truncation rule. Instead of: keep K tokens we say: keep the smallest set of tokens whose cumulative probability reaches p Suppose: A 0.45 B 0.25 C 0.15 D 0.08 E 0.04 F 0.02 G 0.01 With: top_p = 0.80 we accumulate: A 0.45 A+B 0.70 A+B+C 0.85 So the nucleus is: {A, B, C} The remaining tokens disappear. Notice the important difference from top-k. If the distribution becomes very concentrated: A 0.95 B 0.02 C 0.01 ... then top_p=0.95 may retain only a handful of candidates. If the model becomes uncertain: A 0.15 B 0.14 C 0.13 D 0.12 E 0.11 ... the nucleus can expand substantially. That is the central idea: Top-p adapts the candidate set to the model's uncertainty. Holtzman et al. found that nucleus sampling could produce distributions much closer to human text than several conventional decoding methods, while avoiding both the blandness of highly deterministic decoding and the incoherence of unconstrained sampling. This is why a setting like: temperature = 0.8 top_p = 0.95 became a common practical combination. Temperature shapes the distribution. Top-p decides how far into the tail you are willing to go. That distinction is fundamental. 5. Min-p: scale the cutoff to the best token Min-p is a newer idea, proposed by Minh Nguyen and colleagues and published at ICLR 2025. It starts with a criticism of top-p: Top-p looks at cumulative probability mass, but sometimes what matters more is the relative quality of each candidate compared with the model's best candidate. Suppose the best token has probability: P_max Min-p keeps tokens satisfying approximately: P(token) >= min_p * P_max So if: P_max = 0.60 min_p = 0.1 the threshold becomes: 0.1 * 0.60 = 0.06 Any token below 0.06 is discarded. Now consider a much less confident model: P_max = 0.12 The threshold becomes: 0.1 * 0.12 = 0.012 The candidate set expands. This gives min-p a nice conceptual interpretation: Keep tokens that are not too far below the model's current favorite. Compare the three strategies: top-k: keep exactly K tokens top-p: keep enough tokens to cover probability mass p min-p: keep tokens above a fraction of the best token's probability They are solving similar problems from different angles. A useful concrete example: Token Probability A 0.50 B 0.20 C 0.10 D 0.08 E 0.04 F 0.03 G 0.02 H 0.01 With: top_k = 4 you get: A B C D With: top_p = 0.80 you also get: A+B+C = 0.80 so roughly: A B C With: min_p = 0.10 the threshold is: 0.10 * 0.50 = 0.05 so: A B C D survive. The important practical point is that min-p is not simply "top-p but newer." It encodes a different assumption about what makes a token trustworthy. There is also a useful lesson here for engineers: newer sampling methods are not automatically better sampling methods. The original min-p paper reported improvements in quality/diversity tradeoffs, particularly at higher temperatures, but a later 2025 critical re-analysis challenged several of those conclusions. That makes min-p interesting precisely because it is an active research area rather than a solved problem. 6. Repetition penalty: change the score of tokens you've already used Sampling filters are mostly concerned with the current distribution. Repetition penalty introduces history. Suppose the model's current logits are: " the" 8.0 " a" 7.5 " cat" 6.8 " dog" 5.9 and " the" has already appeared many times. A repetition penalty says, approximately: If a token has already appeared, make it less attractive. The classic approach associated with CTRL applies a multiplicative transformation to previously seen tokens. In the common sign-aware implementation: if z > 0: z' = z / penalty if z