Ask ten developers what temperature does in an LLM and you'll get ten vague answers involving "creativity." The real mechanism is simpler and more useful to understand — and getting it right is the difference between a model that reliably extracts JSON and one that hallucinate-explains its own hallucinations.
The Simple Version
Temperature controls how random the model's word choices are.
An LLM generates text one token at a time. At each step, it computes a probability for every possible next token. Temperature reshapes those probabilities before the choice is made:
- Low temperature (toward 0): The model almost always picks the highest-probability token. Output becomes predictable, focused, and repetitive.
- High temperature (toward 2.0): Probability is spread out across more tokens. Output becomes varied, surprising, and — past a point — incoherent.
A useful analogy: temperature is the difference between a careful librarian and an improvising poet sharing the same brain. Same knowledge, different selection discipline.
What's Actually Happening Under the Hood
Technically, temperature is a divisor applied to the logits (the raw scores) before the softmax that turns them into probabilities:
new_logits = logits / T- When T < 1, high scores get relatively higher — the rich get richer. The top token dominates.
- When T > 1, scores get compressed — the underdogs gain ground. Unlikely tokens get real chances.
- When T = 1, the model samples from its raw learned distribution, unmodified.
At T = 0 (greedy decoding), randomness is removed entirely: the single highest-scoring token is always chosen. This is why temperature-0 outputs are deterministic per run on the same hardware, but can still differ across deployments — floating-point non-determinism and batching effects can flip which token "wins" when scores are close.
Here's a concrete illustration. Suppose after the phrase "The capital of France is", the model's top token probabilities are:
| Token | Probability |
|---|---|
Paris | 0.86 |
the | 0.06 |
located | 0.03 |
a | 0.02 |
| ... | ... |
At T=0, you get "Paris" every time. At T=1, you get "Paris" 86% of the time — and something else 14% of the time, which is a lot of wrong answers for a factual prompt. At T=0.2, the distribution sharpens so dramatically that "Paris" wins ~99% of the time. That's the whole trick: temperature doesn't change what the model knows, it changes how strictly the model obeys its own knowledge.
The Default: Why 0.7?
Most APIs default to temperature 0.7–1.0. That range is a compromise: varied enough that conversations don't feel robotic, grounded enough that answers stay on-topic. It's tuned for chat — the most common use case — not for your use case.
Which is exactly the problem. Defaults are averages, and your task isn't. A data-extraction pipeline running at the chat default is quietly rolling dice on every field; a novel-writing session at the same setting is leaving most of the model's range on the table.
When to Use Low Temperature (0–0.3)
Use low temperature when there is one right answer and you want the model to find it consistently:
- Data extraction — pulling fields from invoices, emails, or documents into JSON
- Classification — sentiment labels, routing, tagging
- Code generation — especially anything that must compile or parse
- Math and formal reasoning — you want the most probable chain, not a creative one
- Evaluation harnesses — when grading other models' outputs, keep the judge consistent
- Translation — fidelity beats flair
Rule of thumb: if you'd be annoyed by "creative" variations in the output, temperature should be near 0.
When to Use High Temperature (0.8–1.2)
Use high temperature when many good answers exist and sameness is the failure mode:
- Brainstorming — idea generation, name suggestions, "give me 20 angles on this"
- Marketing copy variations — A/B testing headlines and CTAs
- Creative writing — fiction, poetry, roleplay
- Data augmentation — generating synthetic variations of training examples
- Avoiding mode collapse — in agent loops, a little randomness prevents every retry from repeating the same failed path
Past ~1.2, coherence decays quickly on most models. Past 1.5, you're mostly generating avant-garde poetry.
Temperature vs. Top-p (and Friends)
Temperature isn't the only sampling dial, and it interacts with the others:
| Parameter | What it controls | Typical range |
|---|---|---|
temperature | Overall randomness (logit scaling) | 0 – 2 |
top_p | Nucleus size — keep only tokens covering the top p% of probability mass | 0.1 – 1 |
top_k | Keep only the k most likely tokens | 1 – 100 |
frequency_penalty | Penalize tokens by how often they've appeared | -2 – 2 |
presence_penalty | Penalize tokens that have appeared at all | -2 – 2 |
The standard advice — from OpenAI's own docs — is don't tune temperature and top_p together; change one, measure, then decide. For most practical work, temperature alone is enough, and top_p stays at 1.
One nuance worth knowing: temperature and topp fail differently. Very low temperature can cause looping (the model repeatedly picks the same safe token), while very low topp strictly cuts the tail and tends to avoid loops but can feel flat. If you see repetition loops at T=0, a small frequency penalty (0.1–0.3) often fixes it more gracefully than raising temperature.
A Real Example
Same prompt, same model, three temperatures. Task: complete "The three most important factors in choosing an API are ___".
T = 0.2:
...reliability, documentation, and cost.
Every run, near-identical. Boring. Correct.
T = 0.7:
...reliability, latency, and whether the docs match reality.
Slightly different each run. Still sensible.
T = 1.4:
...trust, the smell of the homepage, and whether the founder has ever been paged at 3 a.m.
Memorable! Not what you want in a pipeline.
Common Mistakes
- Leaving chat defaults on pipeline work. Running extraction at temperature 0.7 injects unnecessary variance into systems that need determinism. This is the single most common temperature bug in production.
- Assuming T=0 means fully deterministic. It doesn't — near-tied logits plus hardware nondeterminism can still produce different outputs across runs and providers. If you need true determinism, cache; don't rely on greedy decoding.
- Cranking temperature to fix bad prompts. If outputs are dull or repetitive at T=0.7, the fix is usually the prompt (or the model), not more randomness.
- Ignoring model-specific differences. Reasoning-style models often apply their own sampling internally during thinking phases; extreme temperatures can interact with that in surprising ways. Test per model.
- Ignoring it entirely. The parameter exists on every major API for a reason — it's the cheapest quality lever you have.
Temperature Across Models: Does It Mean the Same Thing Everywhere?
Mechanically, yes — every major provider applies the same logit-scaling math. Practically, no — models have differently shaped output distributions, so T=0.8 on one model can feel like T=1.1 on another. A highly confident model (sharper distributions) stays coherent at higher temperatures; a more uncertain model falls apart sooner.
This matters when you route the same prompt across multiple providers — a common pattern for cost optimization or failover. Your extraction pipeline might be rock-solid at T=0.1 on one model and occasionally weird at T=0.1 on another. The fix is to treat sampling parameters as part of your per-model config, not a global constant. If you're running multi-model setups, tools that let you set parameters per model — like the configuration options described at qubax.ai/docs — make this painless.
Key Takeaways
- Temperature rescales the model's token probabilities: low = focused, high = varied
- 0–0.3 for extraction, classification, code; 0.8–1.2 for ideation and creative work
- The 0.7 default is a chat compromise, not a recommendation for your task
- Don't co-tune with top_p; and T=0 isn't a determinism guarantee
- Same temperature behaves differently across models — configure per model, not globally
- The API you call should expose the dial — check that yours does at qubax.ai/models
FAQ
What is temperature in AI models?
Temperature is a setting that controls how random the model's output is. Low values make output focused and predictable; high values make it more varied and creative.
What temperature should I use for coding?
0 – 0.2. Code must compile and parse; you want the model's most probable continuation, not a creative one.
What temperature is best for writing?
0.7 – 1.0 for general prose and marketing copy; up to 1.2 for fiction and brainstorming, watching for coherence loss.
Is temperature 0 deterministic?
Not fully. T=0 picks the top token every time, but floating-point non-determinism, batching, and provider-side changes can still cause different outputs across runs or deployments.
Does temperature change what the model knows?
No. It only changes how selections are made from what it already knows. High temperature can't add knowledge — it can only make the model more willing to pick unusual (including wrong) tokens.
Should I set temperature or top_p?
Pick one. OpenAI's documentation recommends adjusting either temperature or top_p, not both, since their effects multiply in unpredictable ways. Most teams get everything they need from temperature alone.
Do all AI APIs support temperature?
All major text APIs do. If you're evaluating providers, verify sampling parameters are exposed and honored — browse available models and their capabilities at qubax.ai/models.
