Batch-invariant inference: Why the same prompt can give different answers, and how to stop it

Batch-invariant inference is a way of running a language model so that a given prompt always produces the same output, no matter how many other requests the server is handling at that moment. It targets a quiet flaw: even with randomness switched off, the same question can get different answers on different runs.

The idea became widely known after a September 2025 write-up by Horace He and colleagues at Thinking Machines.

The same question, asked 1,000 times, gave 80 different answers

Imagine asking a chatbot one question twice, with the temperature setting at 0. Temperature controls how much randomness the model adds when it picks the next word, and 0 means always pick the most likely one. You would expect identical replies. Often you do not get them.

Thinking Machines tested this with Qwen3-235B-A22B-Instruct-2507, an open model. They asked it to "Tell me about Richard Feynman" 1,000 times at temperature 0, with 1,000 tokens per answer. They got 80 unique completions. The texts matched for the first 102 tokens, then split at token 103: 992 continued with "Queens, New York," and 8 with "New York City."

With batch-invariant kernels switched on, all 1,000 completions were identical, according to the same write-up.

Why adding numbers in a different order changes the result

The root cause is small. Computers store decimals with limited precision, so addition is not perfectly associative. In standard double-precision arithmetic, (0.1 + 0.2) + 0.3 gives 0.6000000000000001, while 0.1 + (0.2 + 0.3) gives 0.6. Same numbers, different grouping, slightly different answer.

A language model performs billions of additions. If the order changes, tiny differences appear. Usually they are harmless. But when two candidate words are nearly tied, a tiny difference can flip the choice. After that, the rest of the reply follows a different path.

The real culprit is server load, not random timing

A popular explanation blames parallel GPU threads finishing in unpredictable order. Thinking Machines argued this is incomplete. In their account, a single forward pass of the model gives the same result every time for the same inputs. The problem is that the result depends on batch size.

A batch is a group of requests processed together, because GPUs are efficient when they work on many things at once. Think of a kitchen that cooks several orders in one pan. Depending on how many orders are in the pan, the cook may switch method. Your dish gets prepared slightly differently on a busy night than on a quiet one.

Software kernels (the small programs that run the math on a GPU) do exactly this. They choose how to split and add up numbers based on batch size. Server load changes unpredictably, and the batch size changes with it, so from one user's view the output looks random.

Batch invariance means each request's calculation uses a fixed order of operations whatever the batch size. The authors' requirement is simple to state: for every output value, the partial sums must be combined in the same sequence, whether the batch holds one request or hundreds.

Three places where the fix has to be made

According to Thinking Machines, three parts of a transformer model needed special kernels:

• RMSNorm, a normalization step: keep one reduction strategy instead of changing it for small batches.
• Matrix multiplication, the core operation: compile one kernel configuration and use it for every input shape.
• Attention, where the model weighs earlier tokens: update the KV cache (the stored record of earlier tokens) before the attention kernel runs, and split the work into fixed-size pieces rather than a fixed number of pieces.

They released a library of these kernels, batch-invariant-ops, and showed it working with vLLM, a widely used open-source inference engine.

Where it is used today, including DeepSeek's new release

vLLM now documents the feature as beta. Setting the environment variable VLLM_BATCH_INVARIANT=1 turns it on. Its documentation lists NVIDIA GPUs with compute capability 8.0 or higher, certain AMD Instinct GPUs and Intel XPUs with Triton support as supported hardware. It names four uses: reproducing framework bugs, debugging model implementations, reinforcement learning with stable rollouts, and testing large systems.

The reinforcement learning case is notable. Thinking Machines reported an experiment in which training with mismatched numerics made reward collapse unless corrected with importance weighting, while exactly matching numerics kept KL divergence at 0. That is the research team's result on one setup, not a general guarantee.

DeepSeek has now built the idea into its own stack. Its newly open-sourced DeepInfer inference framework offers batch-invariant inference and deterministic sampling. According to the project's README, as reported in our coverage, both work on NVIDIA hardware only, not on Huawei Ascend chips. The aim is identical outputs regardless of batch composition, or for identical inputs and sampling settings.

What it costs, and where it still falls short

Reproducibility is not free. A fixed, one-size-fits-all kernel cannot adapt to each batch size. Thinking Machines measured about 20% lower matrix-multiplication performance than NVIDIA's cuBLAS library. In an end-to-end test with Qwen-3-8B on one GPU, 1,000 sequences took 26 seconds with default vLLM, 55 seconds with an unoptimized deterministic version, and 42 seconds after they improved the attention kernel. These figures come from the authors' own test, and results will differ on other models and hardware.

vLLM's documentation also says enabling the feature may slow inference compared with the default mode, and calls this an intentional trade-off. It lists gaps too. Under tensor parallelism (splitting one model across several GPUs), some all-reduce operations whose size is not a multiple of 16 bytes are not yet supported.

Two limits are worth keeping in mind. Identical output on one setup does not mean identical output across different GPU types or software versions, since the numbers can still be computed differently there. And determinism is not accuracy: a model can repeat the same wrong answer every time.

What to watch next

Three things will show whether this becomes standard. First, how much speed the major engines recover as their deterministic kernels mature. Second, whether more providers offer reproducible outputs as an option, which would help teams that need audit trails or stable test suites. Third, how the approach carries over to non-NVIDIA hardware, where DeepInfer's own support is currently missing.