AMD says GLM-5.2 on eight MI355X GPUs now reads 1M tokens 7x faster

AMD says it has raised GLM-5.2 FP8 throughput on eight Instinct MI355X GPUs by up to roughly four times at 64K and 128K inputs, and 7.16x at one million tokens, according to a post on its ROCm blog.

The results come from software changes across the serving stack, not new hardware. AMD measured them with EvalScope, an open benchmark client, rather than a client built into the serving framework.

Peak prefill throughput roughly quadruples at long inputs

The test setup uses SGLang on ROCm 7.2.4 with the zai-org/GLM-5.2-FP8 model, driven through the OpenAI-compatible /v1/completions endpoint. Output length is one token, so the figures describe prefill, the stage where the model reads the prompt. At 64K and 128K, AMD used a 90% shared-prefix setup to mimic servers that reuse cached context.

AMD reports these peaks, all from runs with zero request failures:

• At 64K input, peak throughput rose from 163K tok/s (concurrency 4) to 629K tok/s (concurrency 50).
• At 128K input, it rose from 137K tok/s to 553K tok/s, with the best optimized result at concurrency 32.
• At 1M input and concurrency 1, throughput went from 21,877 to 156,598 tok/s, and average time to first token fell from 45,660 ms to 6,340 ms.

The baseline flattens after concurrency 4 at 64K, while the optimized stack keeps scaling to 50. At the 128K setting the optimized result peaks near concurrency 8 to 32 and then dips slightly at 50.

Context parallelism is the central change

With the usual tensor parallelism on eight GPUs, each GPU holds only eight of GLM-5.2’s 64 attention heads, yet the model’s indexer picks the same set of cache entries for all heads. AMD says that means every GPU loads the same data, and that kernels needing head counts in multiples of 16 force the eight heads to be copied up to 16.

AMD’s fix is prefill context parallelism (CP8), which splits the tokens instead of the heads. Each GPU keeps all 64 heads and handles one-eighth of the tokens, which lets the chip use its native 64-head MLA kernel. AMD added SGLang support so a token on one GPU can still select cache entries held elsewhere.

Other changes include an FP8 KV cache, a single fused Triton kernel for query quantization, tuned MoE communication that cuts the payload per token from 12,288 to 6,408 bytes, tuned GEMM tables in AITER, and EAGLE multi-token prediction with 4 steps and 5 draft tokens.

How far the numbers travel beyond this test

AMD’s own figures need care. They measure prefill on a synthetic random dataset with one output token, so they say little about decode speed or about a mixed production workload. The baseline is a specific SGLang ROCm image dated June 18, so the gain also reflects software that was not yet tuned for this model.

A separate AMD post from November 4 on the same model shows a different kind of workload. For GLM-5.2-MXFP4 on the InferenceX AgentX benchmark, built from 393 reconstructed Claude Code sessions, AMD reported per-GPU peak throughput rising from 10,625 to 18,327 tokens per second. In that test the median request carried about 142,000 input tokens but only 444 output tokens. AMD itself noted that the comparison is between two deployed configurations, not a controlled A/B test. That is a reminder that gains of 4x in prefill tables translate into much smaller gains in end-to-end agent workloads.

AMD also reports accuracy held up. On the full 1,319-question GSM8K test set, the baseline scored 94.16% (1,242 correct) and the optimized stack 94.92% (1,252 correct). One math benchmark is a narrow check for changes to attention and quantization, and the posted numbers have not been independently verified.

What developers can reproduce today

AMD published launch and benchmark scripts (launch_baseline.sh, launch_optimized.sh and run_evalscope.sh) alongside the post. The optimized build ships as a Docker image, sabreshao/sglang:glm_0831, which appears to be a personal registry rather than an official AMD release, and the post does not say when the changes will land in upstream SGLang or AITER. Anyone replicating the results needs eight MI355X GPUs, so this is aimed at data-center operators, not hobbyists.

The work fits AMD’s push to show Instinct GPUs running current open models and agent workloads. Earlier this month the company described PrimusClaw, an open-source agent runtime for its own GPUs, and Anthropic’s roadshow said its model put Claude on AMD chips in three days. The post makes no comparison with Nvidia hardware on this workload, so how the MI355X stacks up on the same test remains open.