Hot AI News

Google’s TPU guide: Set head_dim to 256 and lift a 30B MoE’s MFU

Google's new open-source tpu-codesign guide lists eight design rules for Ironwood and later TPUs, and says tuning alone took an open 30B MoE from 12% to 23% MFU.

Google’s TPU guide: Set head_dim to 256 and lift a 30B MoE’s MFU
Image: Google

Google has released a TPU model design guide, published as the open-source AI-Hypercomputer/tpu-codesign repository. It sets out eight rules for LLMs and mixture-of-experts models on Ironwood (TPU v7x). Its headline claim: systems tuning alone lifted an open 30B MoE from 12% to 23% MFU.

Eight rules, from shape alignment to parallelism

The post, written by Google engineers Gagik Amirkhanyan and Vaibhav Singh, sorts the rules into three groups. Shape rules (1 and 2) carry almost no risk to model quality, and Google says to adopt them always. Architecture choices (3 to 7) usually involve quality tradeoffs and should be checked with ablations. The last group, parallelism (8), needs no change to the model itself.

Chart of training speedup from reshaping attention to head_dim 256 in a Qwen3-30B MoE
Image: Google

The first rule is to make reduction and output dimensions multiples of 256, to match Ironwood’s 256×256 matrix units. Google says the most commonly missed one is head_dim. At 64 or 128, it says, flash-attention products leave the matrix unit at least half idle. In one measured case, reshaping Qwen3-30B MoE attention from 32 x 128 to 16 x 256 kept parameters and FLOPs identical and sped up end-to-end training, with the gain growing at longer context.

The second rule covers dynamic workloads. XLA compiles for exact shapes, so Google recommends padding requests into precompiled buckets and masking. For dropless MoE layers and paged attention, it recommends keeping buffers fixed and passing the ragged structure as small integer arrays through Pallas scalar prefetch.

Chart comparing MoE expert layouts and training throughput on a 63B LatentMoE
Image: Google

The third rule is LatentMoE. It projects routed tokens down to a latent width, typically 2 to 4 times smaller than d_model, before dispatch. Google says that on a 30B, 128-expert MoE, a 768-wide latent cut on-device permute time 2.8x. It also lifted training throughput 14% in tokens per second and freed about 25 GB of activation memory per device. Google adds that standard expert parallelism was slower than plain FSDP in its Ironwood benchmarks.

A second measured result concerns compute intensity. On a 63B LatentMoE across 512 TPU v7x chips, halving the expert count and top-K while doubling expert width raised throughput 57% and held lower loss over 120.8B tokens.

The remaining rules cover decode and the vector unit. They advise sizing designs to real serving concurrency, using multi-token prediction heads for speculative decoding, and keeping vector work hidden inside matmul windows. They also advise mapping parallel axes onto the fastest physical links. Google says offloading collectives to SparseCore cut step time by 15 to 22% in its tpu-recipes benchmarks.

Systems tuning alone lifted an open 30B MoE from 12% to 23% MFU, before any model change.

Advertisements

Newer chips change thresholds, not the logic

According to Google, the same design logic holds on its eighth-generation hardware: TPU 8t, TPU 8i and the two variants it has just announced, TPU 8tx and TPU 8i Max. Only the numeric thresholds move with the chips. TPU 8t and TPU 8i each add native FP4. TPU 8i adds the Boardfly topology, a collectives engine that Google says cuts collective latency by up to 5x, and three times the on-chip SRAM. Google says Boardfly puts every chip in a 1,024-chip pod within seven hops, against 16 on a torus of the same size. Google promises follow-up posts on eighth-generation designs.

Some numbers are explicitly Ironwood-specific. The block-size threshold for block-scaled FP8 matmuls, about 340 on Ironwood, should be re-measured on TPU 8t rather than reused.

Why the 256 rule is harder than it looks

The guide’s core argument is that architectural choices made before training starts set a permanent ceiling on utilization, cost and serving latency. A TPU trades flexible hardware scheduling for wide systolic arrays, so a model whose dimensions do not fill them wastes silicon that no compiler can win back later. The VPU example makes this concrete. Per Google’s figures, Ironwood’s FP8 peak is 4,614 TFLOPS against 19.2 TFLOPS of vector peak, a ratio near 240. Small quantization blocks therefore become vector-bound: in Google’s simulation, block 256 ran 1.2x slower than per-tensor scaling and block 128 ran 2.4x slower.

Google offers one comparison to show alignment pays off. It says published Ironwood baselines show 256-aligned MoEs such as DeepSeek-V3 and Qwen3-235B at 608 to 630 BF16 TFLOP/s per chip, roughly twice the throughput of the unaligned GPT-OSS-120B at 330.

The guide also fits a pattern in Google’s recent TPU publishing. Our earlier report on Tunix’s bit-exact reinforcement learning on TPU described a prototype tested on 64 TPU v5p chips with a public Qwen3-8B checkpoint. That work targets numerical consistency, while this guide targets efficiency. Both are aimed at outside developers building on Google’s accelerators.

Advertisements

What teams can use now, and what is unproven

The guide and companion configs are open source, with MaxText and tpu-recipes among the referenced projects. The post does not mention pricing or access conditions. Teams adapting an existing checkpoint can use the parallelism section without touching the model. Teams designing from scratch can apply the shape rules at essentially no quality risk.

The evidence has limits. The measured results come from Google’s own profiling program. It covers open 20 to 30B MoEs and dense controls, mostly on a single slice and in BF16, with the 512-chip and 1,024-chip ablations as exceptions. Google itself says the trends should transfer but the exact percentages are specific to each configuration. No independent party has reproduced the numbers, and the post gives no figures for TPU 8t, 8i, 8tx or 8i Max beyond architecture details. Several ideas, including LatentMoE, come from other labs’ models, so the main contribution here is the TPU-specific guidance and measurements.

Sign Up for the Daily AI Pulse

One email a day. All the stories that matter.

Alex CarterAI News & Big Tech Correspondent

Alex Carter writes for Aiholics, keeping readers updated on the fast-paced world of AI and Big Tech.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may also like