Google’s TPU guide: Set head_dim to 256 and lift a 30B MoE’s MFU
Google's new open-source tpu-codesign guide lists eight design rules for Ironwood and later TPUs, and says tuning alone took an open 30B MoE from 12% to 23% MFU.

Google has released a TPU model design guide, published as the open-source AI-Hypercomputer/tpu-codesign repository. It sets out eight rules for LLMs and mixture-of-experts models on Ironwood (TPU v7x). Its headline claim: systems tuning alone lifted an open 30B MoE from 12% to 23% MFU.
Eight rules, from shape alignment to parallelism
The post, written by Google engineers Gagik Amirkhanyan and Vaibhav Singh, sorts the rules into three groups. Shape rules (1 and 2) carry almost no risk to model quality, and Google says to adopt them always. Architecture choices (3 to 7) usually involve quality tradeoffs and should be checked with ablations. The last group, parallelism (8), needs no change to the model itself.

The first rule is to make reduction and output dimensions multiples of 256, to match Ironwood’s 256×256 matrix units. Google says the most commonly missed one is head_dim. At 64 or 128, it says, flash-attention products leave the matrix unit at least half idle. In one measured case, reshaping Qwen3-30B MoE attention from 32 x 128 to 16 x 256 kept parameters and FLOPs identical and sped up end-to-end training, with the gain growing at longer context.
The second rule covers dynamic workloads. XLA compiles for exact shapes, so Google recommends padding requests into precompiled buckets and masking. For dropless MoE layers and paged attention, it recommends keeping buffers fixed and passing the ragged structure as small integer arrays through Pallas scalar prefetch.

The third rule is LatentMoE. It projects routed tokens down to a latent width, typically 2 to 4 times smaller than d_model, before dispatch. Google says that on a 30B, 128-expert MoE, a 768-wide latent cut on-device permute time 2.8x. It also lifted training throughput 14% in tokens per second and freed about 25 GB of activation memory per device. Google adds that standard expert parallelism was slower than plain FSDP in its Ironwood benchmarks.
A second measured result concerns compute intensity. On a 63B LatentMoE across 512 TPU v7x chips, halving the expert count and top-K while doubling expert width raised throughput 57% and held lower loss over 120.8B tokens.
The remaining rules cover decode and the vector unit. They advise sizing designs to real serving concurrency, using multi-token prediction heads for speculative decoding, and keeping vector work hidden inside matmul windows. They also advise mapping parallel axes onto the fastest physical links. Google says offloading collectives to SparseCore cut step time by 15 to 22% in its tpu-recipes benchmarks.
Systems tuning alone lifted an open 30B MoE from 12% to 23% MFU, before any model change.
Newer chips change thresholds, not the logic
According to Google, the same design logic holds on its eighth-generation hardware: TPU 8t, TPU 8i and the two variants it has just announced, TPU 8tx and TPU 8i Max. Only the numeric thresholds move with the chips. TPU 8t and TPU 8i each add native FP4. TPU 8i adds the Boardfly topology, a collectives engine that Google says cuts collective latency by up to 5x, and three times the on-chip SRAM. Google says Boardfly puts every chip in a 1,024-chip pod within seven hops, against 16 on a torus of the same size. Google promises follow-up posts on eighth-generation designs.
Some numbers are explicitly Ironwood-specific. The block-size threshold for block-scaled FP8 matmuls, about 340 on Ironwood, should be re-measured on TPU 8t rather than reused.
Why the 256 rule is harder than it looks
The guide’s core argument is that architectural choices made before training starts set a permanent ceiling on utilization, cost and serving latency. A TPU trades flexible hardware scheduling for wide systolic arrays, so a model whose dimensions do not fill them wastes silicon that no compiler can win back later. The VPU example makes this concrete. Per Google’s figures, Ironwood’s FP8 peak is 4,614 TFLOPS against 19.2 TFLOPS of vector peak, a ratio near 240. Small quantization blocks therefore become vector-bound: in Google’s simulation, block 256 ran 1.2x slower than per-tensor scaling and block 128 ran 2.4x slower.
Google offers one comparison to show alignment pays off. It says published Ironwood baselines show 256-aligned MoEs such as DeepSeek-V3 and Qwen3-235B at 608 to 630 BF16 TFLOP/s per chip, roughly twice the throughput of the unaligned GPT-OSS-120B at 330.
The guide also fits a pattern in Google’s recent TPU publishing. Our earlier report on Tunix’s bit-exact reinforcement learning on TPU described a prototype tested on 64 TPU v5p chips with a public Qwen3-8B checkpoint. That work targets numerical consistency, while this guide targets efficiency. Both are aimed at outside developers building on Google’s accelerators.
What teams can use now, and what is unproven
The guide and companion configs are open source, with MaxText and tpu-recipes among the referenced projects. The post does not mention pricing or access conditions. Teams adapting an existing checkpoint can use the parallelism section without touching the model. Teams designing from scratch can apply the shape rules at essentially no quality risk.
The evidence has limits. The measured results come from Google’s own profiling program. It covers open 20 to 30B MoEs and dense controls, mostly on a single slice and in BF16, with the 512-chip and 1,024-chip ablations as exceptions. Google itself says the trends should transfer but the exact percentages are specific to each configuration. No independent party has reproduced the numbers, and the post gives no figures for TPU 8t, 8i, 8tx or 8i Max beyond architecture details. Several ideas, including LatentMoE, come from other labs’ models, so the main contribution here is the TPU-specific guidance and measurements.
Sign Up for the Daily AI Pulse
One email a day. All the stories that matter.
By signing up, you agree to our Terms of Use and acknowledge the data practices in our Privacy Policy. You may unsubscribe at any time.
Alex CarterAI News & Big Tech CorrespondentAlex Carter writes for Aiholics, keeping readers updated on the fast-paced world of AI and Big Tech.



