Non-Deterministic LLM Prompts in 2026: Temperature, Sampling, Seeds, and How to Make Outputs Reproducible

TL;DR

LLMs are not deterministic because the model is probabilistic and most serving stacks are not batch-invariant. The decoder samples each next token from a probability distribution, and even at temperature 0 your request shares a GPU batch whose size changes with server load, which can shift the logits. You can bound non-determinism with temperature, seed and version pinning, then score outputs on semantic equivalence instead of byte-exact match.

Why are LLMs not deterministic?

A language model assigns a probability to every possible next token given the context. A decoder picks one. The math at each step is:

1. Forward pass produces logits (raw scores) for every token in the vocabulary.
2. Temperature divides logits by T.
3. Softmax converts logits to probabilities.
4. Optional top-k or top-p truncation.
5. The decoder samples one token from the resulting distribution.
6. The token is appended to the context and the loop continues.

Even at temperature=0 (where step 5 becomes argmax), step 1 can return slightly different logits on a shared inference server. Most matmul, normalization and attention kernels are not batch-invariant, and because floating-point addition is not associative, a change in reduction order changes the last bits of the logits. In tight contests between two top tokens, this can flip the argmax. In production, additional non-determinism comes from:

• Batch size and server load. Continuous batching in vLLM and dynamic batching in SGLang size your batch by whatever else is in flight, and kernels that are not batch-invariant change their reduction order with it.
• Datacenter routing. The same API call may land on a different GPU type.
• MoE routing. Mixture-of-experts models route tokens to different experts based on batch context.

This is why temperature=0 alone is not enough. You also want a pinned seed, a logged system_fingerprint, a pinned model version and a tolerant evaluator.

Batch invariance and server load

Thinking Machines Lab traced most of this drift to one property in its September 2025 post Defeating Nondeterminism in LLM Inference. The forward pass repeats exactly for the same batch, but most matmul, normalization and attention kernels are not batch-invariant, so your request’s result depends on how many other requests share the batch. Server load sets that batch size, and load rises and falls with other users’ traffic.

In their test, sampling Qwen3-235B-A22B-Instruct-2507 1,000 times at temperature 0 with one prompt produced 80 unique completions; with batch-invariant kernels on, all 1,000 were identical. A 2024 arXiv study, Non-Determinism of “Deterministic” LLM Settings, saw accuracy vary by up to 15% across 10 runs of five LLMs configured to be deterministic.

Sampling parameters, properly explained

Temperature

Temperature T divides each logit by T before softmax. The effect:

• T=0: argmax. The highest-probability token wins. Deterministic on paper.
• T=0.7 to 1.0: typical for chat and content generation. Some randomness, mostly stays on high-probability tokens.
• T>1: flattens the distribution. More creative and more chaotic.

Formula (for token i, logit z_i):

p_i = exp(z_i / T) / sum_j exp(z_j / T)

As T -> 0, the distribution becomes a one-hot at the argmax token. As T -> infinity, it approaches uniform.

Top-k

Top-k truncates the distribution to the K highest-probability tokens, then renormalizes. Typical K values are 40 to 100. Lower K reduces diversity and the risk of rare-token hallucinations; higher K allows more variety.

Top-p (nucleus)

Top-p keeps the smallest set of tokens whose cumulative probability is at least p, then renormalizes. Typical p values are 0.9 to 0.95. Unlike top-k, top-p adapts to how peaky or flat the distribution is at each step.

Combining

Self-hosted engines and many hosted APIs expose temperature and top_p (and sometimes top_k), but Anthropic rejects non-default values on Claude Opus 5.5 and other recent Claude models (any value from API version 2026-10-21), and OpenAI’s GPT-6 models do not support the first two whenever reasoning effort is not none. Where accepted, these are starting points to tune, not provider defaults:

• Determinism-leaning workloads: temperature=0, top_p=1, seed pinned, model version pinned.
• Balanced chat: temperature=0.7 (leave top_p at its default).
• Creative generation: temperature=1.0, top_p=0.95.

If you set temperature=0, top_p has no further effect.

Seed and system_fingerprint

OpenAI’s Chat Completions API still accepts a seed and returns a system_fingerprint, but OpenAI now marks both deprecated and calls seeded sampling best effort. The newer Responses API has no seed parameter. If the fingerprint changes between calls, the backend changed and reproducibility is not guaranteed. Other providers offer model-version pinning and varying seed support; treat seed availability as model-specific and read each provider’s docs before relying on it.

The reproducibility ladder

Use this checklist to bound non-determinism as much as the platform allows.

1. Pin the model version. Call the most specific model ID your provider publishes (a dated snapshot where one exists) instead of a moving alias, and log the model string each response reports.
2. Set temperature=0.
3. Set top_p=1 (no effect if temperature=0, but explicit beats implicit).
4. Pin the seed. Use the same integer across runs.
5. Record the system_fingerprint or pin the backend version. For OpenAI, record and validate system_fingerprint. For self-hosted, pin the vLLM or SGLang version, the model checkpoint hash, the dtype, and the batch configuration.
6. For self-hosted, turn on batch-invariant mode (vLLM VLLM_BATCH_INVARIANT=1, SGLang --enable-deterministic-inference) and accept the throughput cost.
7. For reasoning models, pin the model version and the reasoning effort. Where a model rejects temperature and top_p (GPT-6 when reasoning effort is not none; Claude Opus 5.5 and other recent Claude models; any Claude model on API version 2026-10-21), steps 2 and 3 do not apply.
8. Evaluate with semantic equivalence, not byte equality.

Even at the top of this ladder, you should expect occasional drift. The goal is to make drift rare and to detect it when it happens.

How non-determinism breaks evaluation, and how to fix it

A single-run evaluation of a non-deterministic system is statistically meaningless. The pattern that holds up:

• For each prompt, run N rollouts (typically 3 to 10).
• Score each rollout with your evaluator suite.
• Report mean plus standard deviation plus worst case per metric.
• Plot the distribution; a wide tail means the system is unsafe even if the mean looks good.

For research-grade reproducibility:

• Log model version, sampling parameters, seed, system_fingerprint, dataset version, evaluator version.
• Treat the eval result as a tuple, not a scalar.
• Compare experiments by their full configurations, not by one score.

Future AGI’s Apache-2.0 agent-learning-kit (pip install agent-learning-kit) carries the evaluation library, and the platform adds built-in evaluators such as Groundedness, Detect Hallucination, Instruction Adherence, Tone and Is JSON, plus custom LLM-as-judge evals. You call evaluate per rollout, then aggregate.

from fi.evals import evaluate

prompt = "Summarize the support ticket in one sentence."
context = "Customer cannot log in after the recent password reset email expired."

# Pseudocode: substitute your provider SDK call.
# To measure semantic variance, set a non-zero temperature (for example 0.7) and let each rollout sample differently.
# To measure platform drift at fixed temperature=0, keep the same seed across rollouts and watch for any deltas.
rollouts = []
for i in range(5):
    answer = "<your provider SDK call here>"
    score = evaluate(
        "faithfulness",
        output=answer,
        context=context,
    )
    rollouts.append((answer, score.score))

mean_score = sum(s for _, s in rollouts) / len(rollouts)
worst = min(s for _, s in rollouts)
print(f"Mean faithfulness: {mean_score:.2f}, worst rollout: {worst:.2f}")

For LLM-as-judge in your domain language:

from fi.evals.metrics import CustomLLMJudge
from fi.evals.llm import LiteLLMProvider

prompt = "Summarize the support ticket in one sentence."
answer = "<the model's response for this rollout>"

judge = CustomLLMJudge(
    provider=LiteLLMProvider(),
    config={
        "name": "support-summary-grader",
        "model": "your-pinned-judge-model-version",
        "grading_criteria": "Score 1.0 if the summary names the customer's blocker, else 0.0.",
    },
)
result = judge.evaluate([{"input": prompt, "output": answer}])
score = result.eval_results[0].output

For trace-level observability across agentic stacks (LangGraph, OpenAI Agents SDK, CrewAI, Google ADK, Mastra), the Apache-2.0 traceAI library emits OpenTelemetry spans that record the model name, temperature and top_p on each LLM call, and you can add the seed as a custom attribute. That lets you compare two runs side by side and find which prompt or which sampling configuration is unstable.

from fi_instrumentation import register, FITracer

tracer_provider = register(project_name="non-det-experiment")
tracer = FITracer(tracer_provider.get_tracer(__name__))

with tracer.start_as_current_span("rollout") as span:
    span.set_attribute("seed", 42)
    span.set_attribute("temperature", 0)
    span.set_attribute("model.version", "your-pinned-model-version")
    # ... your LLM call

The Future AGI Agent Command Center is an OpenAI-compatible gateway in front of your providers. Turn on its exact-match cache and non-streaming requests with the same messages, model and temperature get the stored response back, so repeated eval prompts come back identical until the entry expires. Send Cache-Control: no-store on rollouts that measure variance.

Non-determinism compounds in agentic workflows

In a single chat completion, non-determinism shows up once. In a five-step agent run, it can compound:

• Step 1: planner picks tool A or tool B, depending on sampling.
• Step 2: tool A returns one set of results, tool B returns another.
• Step 3: the agent’s next plan depends on what step 2 returned.

The result is that two agent runs from the same starting state can take completely different trajectories. Some of this is unavoidable. To bound it:

• Pin every model call with temperature=0 and a per-step seed where the model accepts them.
• Make tools deterministic where you can: idempotent APIs, pinned retrieval indices, cached web fetches.
• Record every span with OpenTelemetry so you can replay a run.
• Use trajectory-tolerant evaluators: score the final outcome and the path quality, not the exact tool sequence.
• Run N agent rollouts per scenario in your eval suite, just like you would for single LLM calls.

For more on agent observability and evaluation, see our agent observability vs evaluation vs benchmarking 2026 writeup.

Extra non-determinism levers when you self-host

When you run the model yourself (vLLM, SGLang, llama.cpp, Ollama), you control more knobs:

• dtype. bf16 and fp16 are not bitwise equal. Pin one and stick with it.
• batch size. Output depends on batch size unless the engine runs batch-invariant kernels, and continuous batching changes batch size with load.
• Inference engine version. Pin the vLLM or SGLang version; vLLM documents reproducibility only on the same hardware and vLLM version.
• GPU type. A100, H100, and L40S may produce slightly different logits at the bit level due to kernel implementation differences.
• MoE routing. For mixture-of-experts models, batch context affects routing. Batch-invariant mode covers MoE models too; vLLM lists DeepSeek-V3 and DeepSeek-R1 among its tested models.

For research reproducibility, the standard practice is to ship a config file with the inference engine version, GPU type, dtype, batch size, sampling parameters, seed, and model checkpoint hash. traceAI records the model name and sampling parameters on each LLM span; attach the rest as custom span attributes with FITracer.

Are there any deterministic LLMs?

Self-hosted engines can now serve open-weight models deterministically. For vLLM deterministic inference, start the server with VLLM_BATCH_INVARIANT=1, which makes output independent of batch size and request order; vLLM still labels batch invariance a beta feature and notes it may reduce performance. Its docs list NVIDIA GPUs with compute capability 8.0 or higher, AMD Instinct GPUs and Intel XPUs.

SGLang offers the same idea through --enable-deterministic-inference, built on Thinking Machines Lab’s batch-invariant operators and limited to the FlashInfer, FlashAttention 3 and Triton attention backends. OpenAI marks its own seed deprecated and best effort, so on a hosted API measure the variance with N rollouts instead of assuming it away.

Common questions and pitfalls

”I set temperature to 0 and still get different answers.”

Likely causes: the model version drifted (you called a moving alias, not a pinned model ID), the backend system_fingerprint changed, or server load changed your batch size. Newer models add one more: GPT-6 does not support temperature whenever reasoning effort is not none, and Anthropic rejects a non-default temperature on Claude Opus 5.5 and other recent models, so there is no temperature 0 to set.

”My evaluation scores swing 10 percent between runs.”

You are scoring a non-deterministic system with a single rollout. Run 5 to 10 rollouts per prompt, report mean plus standard deviation, and pin everything else (model, seed, sampling params, evaluator).

”I want bitwise reproducibility for compliance.”

You are likely going to need self-hosting with batch-invariant mode on vLLM or SGLang, a fixed GPU type and engine version, and acceptance of the throughput cost. Even then, validate with a determinism eval before claiming reproducibility.

”Should I use temperature=0 in production?”

For factual, instruction-heavy, or extraction tasks: yes. For creative content, summarization with style variation, or open-ended chat: a small non-zero temperature (0.3 to 0.7) is usually better because it avoids degenerate repetition and reads more naturally.

Shipping LLM features that live with variance

Same prompt, different answer is the normal state of a shared API, and it is manageable once you can see it. Pin what the provider lets you pin, turn on batch-invariant mode where you host the model, and treat every eval score as a mean with a spread, so a prompt change ships only when it beats the variance.

See our LLM evaluation guide and LLM prompt format patterns for the wider playbook.

Measure your own run-to-run variance with Future AGI evaluations: start free.

Sources

• OpenAI Python SDK, Chat Completions request parameters (seed and top_p notes)
• OpenAI API deprecations (dated model snapshots and shutdown dates)