Ask a model the same question twice and you often get different answers. That's not a bug — it's sampling, and a few settings control how much variation you get.
How models pick words
At each step, a model doesn't output one certain word — it produces a probability distribution over many possible next tokens. Sampling is how it chooses from that distribution. If it always picked the single most likely token (greedy decoding), output would be deterministic but often bland and repetitive. Instead, models sample, introducing controlled randomness.
The knobs
- Temperature scales how "adventurous" the sampling is. Low temperature (near 0) makes the model favor the most likely tokens — focused, consistent, sometimes repetitive. High temperature flattens the odds — more creative and varied, but more prone to going off the rails.
- Top-p (nucleus sampling) restricts choices to the smallest set of tokens whose probabilities add up to p — a dynamic way to cut off unlikely options.
Low temperature is a careful accountant; high temperature is a brainstorming poet. Pick the one your task needs.
Choosing settings
For factual, structured, or code tasks where you want consistency, use low temperature. For creative writing or brainstorming where variety helps, raise it. Note that low temperature reduces variation but never guarantees identical output, and some newer models restrict or remove these parameters in favor of prompting.
Why it matters for apps
If your AI feature needs reliable, repeatable output — extraction, classification, code — high randomness is your enemy; keep temperature low and consider structured outputs. If it needs fresh, varied responses, dial it up. Many production bugs ("why is the answer different every time?") trace straight back to sampling settings. Understanding these knobs turns unpredictable output into a deliberate choice.
