Hot AI News

Google’s Tunix makes TPU reinforcement learning bit-exact between sampler and trainer

Google says its Zero-TIM prototype in the open-source Tunix library removes train-inference mismatch on TPU, tested on 64 TPU v5p chips with a Qwen3-8B checkpoint.

Google’s Tunix makes TPU reinforcement learning bit-exact between sampler and trainer
Image: Google

Google engineers say they have removed a hidden numerical flaw from reinforcement learning on TPUs. In a post on the Google Developers Blog, they describe Zero-TIM, a prototype in the open-source Tunix library. It makes the sampler and the trainer assign bitwise-identical logprobs to the same token.

Three code paths that must agree to the last bit

In LLM reinforcement learning, the model is the policy. Training is only truly on-policy if the weights that generate an action also score it. Production stacks run sampling and training through different execution paths. The sampler records a logprob during incremental decoding, and the trainer recomputes it in batched forward-backward passes. Even with identical weights, the two can disagree. Google calls this a hidden off-policy gap that compounds over multi-turn trajectories.

Diagram of bitwise logprob comparisons between sampler decode, serving prefill and trainer forward
Image: Google

The cause, according to the post, is floating-point arithmetic. Addition is not associative, so a different reduction order changes where rounding happens. The effect is stronger in bfloat16. In the authors’ stack, sampler decode and trainer forward used different token-array shapes and forward implementations, so reduction order and intermediate precision were not aligned.

Google’s fix is a contract across three paths: sampler decode, serving prefill and trainer forward. All three must produce identical raw floating-point bytes for every sampled action, with no tolerance. The prefill path acts as a middle checkpoint. It shows whether a mismatch arises inside the serving engine or across the serving-training boundary.

Chunked differentiation connecting forward chunks through KV-cache state and reverse passes through cache gradients
Image: Google

The recipe has four parts:

  • Batch-invariant Pallas kernels fix accumulation order inside each tensor-parallel shard, accumulating in float32 with explicit output casts.
  • Cross-shard reductions use ppermute to move partial outputs, and every rank adds them in the same source-rank order.
  • Decode, prefill and trainer share one log-softmax construction and score the token that was actually sampled.
  • Custom vector-Jacobian products keep the serving kernels for the forward pass and supply separate backward rules, with --xla_allow_excess_precision=false set through XLA_FLAGS.

Long trajectories are processed in fixed-size chunks that carry the paged KV cache forward. Cache gradients link the reverse passes.

Google's Tunix makes TPU reinforcement learning bit-exact between sampler and trainer
Image: Google
Advertisements

A 200-step FrozenLake test with one seed

Google verified the approach on 64 TPU v5p chips in a DP8xTP8 mesh. The benchmark ran 200 steps on the synthetic FrozenLake environment, with grids from 2×2 to 9×9 and up to five interaction turns. It used a public Qwen3-8B checkpoint, a GSPO-token loss and leave-one-out advantages. Three setups were compared: standard training, token-level truncated importance sampling (TIS) and Zero-TIM. Zero-TIM also preserves sampled token IDs across turns.

The authors report that Zero-TIM recorded a mean absolute sampler-trainer logprob difference of exactly zero at every logged step. Standard and TIS runs kept nonzero differences. TIS reweights the contribution of each sample to the update but, in Google’s words, does not remove the underlying mismatch. Zero-TIM reached higher observed solve rates. The runs are single-seed, and the post gives no final solve-rate numbers in its text, only endpoint labels in a figure.

The code is a prototype, not a finished product. Google links a preview of the implementation in the Tunix repository and says it is extending support to linear-attention and Mixture-of-Experts architectures. Teams already on JAX and Cloud TPU can read the kernels and adapt the patterns. The post makes no claim about GPU stacks or about other TPU generations.

Catching up with GPU-side work, with a speed question

The announcement follows work on GPUs. The vLLM team published a bitwise-consistent on-policy RL setup with TorchTitan in November 2025. By the vLLM team’s account, it reused vLLM’s batch-invariant forward kernels in the trainer and added simple backward passes. The same post reports the bitwise run was 2.4x slower than the non-bitwise case, and that the setup still kept two copies of the model code. Google cites that effort and RadixArk’s Miles as proof that bit-exact alignment is becoming a standard requirement.

That comparison shows the open question. Google’s post does not report training throughput or the cost of fixed accumulation order and custom kernels on TPU. Without those figures, readers cannot tell whether exactness comes cheap here. The evidence is also narrow: a toy game, one model and one seed. It does not show gains on the long-horizon agent tasks where drift would matter most.

The work also fits Google’s wider silicon story. Accelerator vendors compete on large training runs, as shown when Vera Rubin cut DeepSeek-V3 training to 4 minutes in its MLPerf debut. Tooling that makes post-training stable on TPUs matters to customers as much as raw speed. Google is putting that tooling in an open library rather than keeping it internal. Its claims are its own and have not been independently reproduced.

Sign Up for the Daily AI Pulse

One email a day. All the stories that matter.

Daniel ReedAI Research, Safety & Ethics Analyst

Daniel Reed currently works as an AI Research, Safety & Ethics Analyst at Aiholics, writing about how changes in artificial intelligence are affecting and will affect scholarship,…

Leave a Reply

Your email address will not be published. Required fields are marked *

You may also like