LLM Sampling Parameters Explained: Temperature, Top-p, and Friends

The parameters panel of any LLM playground presents a row of sliders, temperature, top_p, penalties, whose names suggest precision engineering and whose tooltips rarely explain what is being engineered. Most developers meet them in a moment of frustration: output too bland, too erratic, or mysteriously different between two identical runs, followed by folklore-driven slider adjustment. The folklore is a shame, because the underlying mechanics are simple enough to hold in your head, and holding them makes the sliders boring in the best way.

All of these parameters govern one small stage of generation. A language model, at each step, scores every token in its vocabulary as a candidate for the next position; sampling parameters decide how a single winner is drawn from those scores, over and over until the response ends. Nothing in this post changes what the model knows or how it reasons. It changes how adventurous the draw is, and that turns out to matter more for some tasks than others.

First, untangle the word "parameter"

Before the sliders, one piece of unavoidable vocabulary hygiene: the word "parameter" means two entirely unrelated things in this field, and both appear on the same model page.

When a model is called 40B or 100B, the B counts billions of parameters in the first sense: the learned numeric values inside the neural network itself. Training is the process of nudging those billions of numbers, over weeks of GPU time, until the network predicts text well; what a model "knows" and how capably it reasons live entirely in them. You will also hear these called weights, and for practical purposes the terms are interchangeable: weights are the overwhelming majority of a network's learned values, so "the weights" and "the parameters" refer to the same object, the giant block of numbers that is the model. Parameter count is the bluntest single indicator of capability and of cost: more parameters generally means more capacity, but also more memory to hold them and more compute per token, which is why bigger models are slower and priced higher. One naming wrinkle worth knowing: mixture-of-experts models advertise two counts, total and active, so a name like 235B-A22B means 235 billion parameters stored, of which about 22 billion participate in any single token. Our guide to reading model names covers the rest of the naming scheme.

"Open weights" follows directly from this: it means the provider publishes the trained parameter values for download, so anyone can run the model on their own hardware, inspect it, or fine-tune it. Llama, Qwen, DeepSeek, and Gemma are open-weight families; GPT-5 and Claude are closed, their parameters never leave the provider and you interact only through an API. Note that open weights is a narrower claim than open source: the numbers are published, but the training data and training code that produced them usually are not. So no, open weights and parameters are not the same kind of thing: parameters are what every model has, open weights describes whether yours are allowed to see them. (Storing those numbers at lower precision to shrink them is quantization, a separate serving decision.)

Sampling parameters, the subject of this post, are the second sense and share nothing with the first except the word. They are not learned, not stored in the model, and not part of what it knows. They are per-request settings in the API call, knobs on the drawing procedure described above, free to change between one request and the next. A 7B model and a 700B model expose exactly the same set. With that distinction in place, "parameters" below always means these request-time knobs.

The one mental model: a distribution being reshaped

Picture the model mid-sentence in "The capital of France is". The score for "Paris" towers over everything; "located", "a", and thousands of others trail far behind. Converted to probabilities, this is a sharply peaked distribution, and almost any sampling configuration will pick "Paris". Now picture "My favorite city is": dozens of tokens are genuinely plausible, the distribution is flat, and the sampling configuration decides everything about how the sentence continues.

Every parameter below is a different way of reshaping or truncating that per-step distribution before the draw. Sharp distributions are barely affected by any of them; flat ones are where the settings live or die. This is why the same temperature feels deterministic on factual prompts and lively on open-ended ones.

Temperature: the sharpness dial

Temperature divides the model's raw scores (logits) before they are converted to probabilities through the softmax function. Dividing by a number below 1 stretches the gaps between scores, so the leaders pull further ahead: probability concentrates, output becomes consistent and conventional. Dividing by a number above 1 compresses the gaps: the distribution flattens, unlikely tokens get real chances, and output becomes varied, then meandering, then broken as the setting climbs past ~1.5.

Temperature 0 is conventionally the switch for greedy decoding: skip sampling entirely and always take the top-scored token. It is the right default for tasks with verifiable answers, code, extraction, classification, with the caveat that greedy selects the most probable answer, not the correct one. A model that believes a wrong thing believes it confidently at every temperature.

Top_p and top_k: trimming the tail

A flat distribution's problem is its tail: thousands of tokens, each individually unlikely, collectively holding enough probability that the sampler occasionally picks something absurd. Truncation parameters cut the tail off before the draw.

top_k is the blunt version: keep only the k highest-probability candidates. Its weakness is that the right k depends on the step: 40 candidates is too many when the model is certain and too few when the continuation is genuinely open.

top_p, nucleus sampling, fixes that by keeping the smallest set of tokens whose cumulative probability reaches p. At top_p: 0.9, a confident step might sample from 2 tokens and an open step from 200; the pool adapts to the model's own certainty. This adaptivity is why nucleus sampling became the standard truncation on hosted APIs and why the usual advice, stated in OpenAI's API reference, is to steer with either temperature or top_p rather than both: they compose multiplicatively, and the combined effect is hard to reason about or reproduce.

Open-source serving stacks such as vLLM expose a newer relative of these, min_p, which keeps tokens whose probability is at least some fraction of the top token's. It behaves like an adaptive top_k, holds up better at high temperatures, and is worth knowing about if you serve open-weights models, though hosted APIs mostly do not expose it.

Penalties: discouraging repetition

Repetition is a failure mode of likelihood itself: once a phrase appears, re-using it is often the locally most probable continuation, and the model can circle. Two parameters push back by docking the scores of tokens that have already appeared. frequency_penalty scales with how often a token has occurred, so the tenth repetition is punished harder than the second; presence_penalty is a flat, one-time deduction that mildly encourages introducing new tokens instead.

Modern instruction-tuned models loop far less than their ancestors, so the right default for both is zero. They earn their keep in long generations that circle, with the standard warning that aggressive values degrade text that legitimately needs repetition, code identifiers being the classic casualty: penalize user_id for appearing five times and the model may creatively rename it on the sixth.

Bounds and reproducibility: max_tokens, stop, and seed

Two parameters bound the output rather than shape it. max_tokens caps the length of the completion and is a cost-control and safety cap, not a target; when generation hits it, the response's finish_reason reads length and the text simply stops, usually mid-sentence. stop sequences end generation the moment a specified string appears, useful for delimiting structured formats. Both matter more than they look: a surprising fraction of "the model gave an incomplete answer" reports are length finishes nobody checked for, a failure mode we cover from the API side in what is an LLM API.

seed aims at reproducibility: fix the randomness so identical requests produce identical output. Providers describe this as best-effort, and the qualifier is honest. Production inference batches concurrent requests together, floating-point arithmetic is not associative so batch composition perturbs the numbers, and Mixture-of-Experts models add routing variability, which is why even temperature 0 does not guarantee bit-identical replies across runs. A seed makes debugging materially easier; it is not a foundation to build correctness guarantees on.

Reasoning models changed the deal

The sliders above assume the model exposes them, and the newest model class often does not. Reasoning models generate an internal chain of thought before the visible answer, and several providers fix or ignore classical sampling parameters for these models entirely, replacing them with a reasoning_effort control that governs how much thinking precedes the answer. The intuition transfers poorly: the lever is no longer "how adventurous is each token draw" but "how much deliberation is purchased before answering", with the cost implications we worked through in the instruct-versus-thinking explainer.

The practical consequence is a check that belongs in any tuning session: confirm which parameters the specific model actually honors, from the provider's documentation or the model's catalog entry, before attributing an output change to a slider that may be silently ignored.

Defaults worth trusting, and when to deviate

A working configuration table, by task, with the reminder that these are starting points rather than laws:

The deeper advice is about tuning order. Sampling parameters are the last knob, not the first: they cannot add knowledge or capability, only redistribute the model's existing tendencies. Output that is wrong wants a better prompt or grounding; output that is wrong in shape wants structured outputs; output that is weak wants a different model, which is a larger lever than any slider on the panel. When the content is right and only the variability is off, that is the moment the sliders earn their place.

Seeing the parameters act is faster than reading about them: run one open-ended prompt at temperature 0.2 and 1.0 side by side and the whole post condenses into one comparison. The playground exposes these controls across models, which also makes the model-versus-slider comparison, often the more decisive one, a single click.