GLM-5.3-Flash on MI300X: Switching parallelism doubles long-context throughput

AMD’s ROCm team has published a serving guide for GLM-5.3-Flash on a single eight-GPU MI300X node. The key finding: tensor parallelism stops scaling after 76 long sessions, and data plus expert parallelism pulls ahead, reaching 2.04x the throughput at 256 concurrent requests.

Only 11 of 45 layers keep a growing cache

GLM-5.3-Flash is a sparse mixture-of-experts model with 320 billion parameters, about 18 billion of them active per token, and a maximum context of 1,048,576 tokens. AMD serves it at up to 131,072 tokens. The FP8 checkpoint takes about 306 GB, roughly a fifth of the node’s 1,536 GB of HBM, so the rest goes largely to the KV cache.

Of the model’s 45 layers, 34 use Kimi Delta Attention, a linear attention whose state does not grow with the prompt. Only the other 11 use sparse Multi-head Latent Attention and store a per-token cache. Each cached row is 512 values wide, because the model has no RoPE half, unlike DeepSeek’s 576-value row.

The cache then decides which parallelism strategy wins. On TP8, the latent cache is copied onto every GPU, so the node’s pool is that of one GPU: 9,841,634 BF16 tokens. A 96k-input, 32k-output request uses 128,000 of them, so the server holds 76 such sessions. On DP8+EP8, each of eight ranks keeps its own requests’ cache, giving 56,564,848 tokens, 5.75x more, or 441 sessions.

TP8 wins alone, DP8+EP8 wins in a crowd

With a single request, TP8 is faster on every shape tested. At 96k/32k it delivers 78.74 output tokens per second against 49.89 for DP8+EP8, a ratio of 0.63x for the latter. On short 1k/1k traffic the gap closes to 0.95x at concurrency 256 but never reverses.

Medium requests cross over earlier than the cache would force. On 8k in and 1k out, DP8+EP8 leads from concurrency 128 (1.19x) and reaches 1.93x at 1,024. For 96k/32k, TP8 still leads at concurrency 64 with 2,105 tok/s against 1,890. At 128 the order flips, 2,085 for TP8 against 3,218, and at 256 DP8+EP8 posts 4,719 against 2,317.

AMD says differences of about 3 percent or less are within run-to-run noise, and each point is a single run.

FP8 cache helps only after BF16 runs out

Storing the cache in FP8 halves bytes per token. Below the BF16 ceiling it changed throughput by less than 2 percent. Past it, gains appeared: +42.7% on TP8 at concurrency 128 and +25.8% on DP8+EP8 at 512. Both narrow once the FP8 pool also fills, to 21.9% and 21.3% at the next point. AMD reads this as a capacity effect, not a bandwidth one.

The setup is not turnkey. The results use vLLM 0.31.0 with ROCm 7.2.3 and three patches: two vLLM changes (#59412 for long-context retrieval, #57134 for FP8 cache) built into a custom Docker image, plus an AITER fix mounted at launch. The FP8 path exists because ROCm’s sparse-attention kernels expect a 576-value row. The post includes a benchmark script, and says it measured throughput, not accuracy, so FP8 cache should be validated on each workload.

A different AMD route than the MI355X work

The approach differs from AMD’s other recent long-context result. Our report on GLM-5.2 on eight MI355X GPUs described context parallelism over SGLang, which AMD said lifted prefill 7.16x at 1M tokens, and noted that the optimized image came from a personal registry rather than an official release. That test measured one-token outputs, so prefill only.

The MI300X post is the opposite: decode-heavy, with 32k generated tokens per request, on older hardware, using the stock vLLM image plus public patches. The two are not comparable, but together they show AMD arguing that memory layout, not raw compute, sets the limit on agent-style loads. The same week, we reported Anthropic’s claim that Claude was ported to AMD GPUs in three days, which adds to the pressure on AMD’s software stack.

The source leaves gaps. All numbers come from AMD, on random-token prompts rather than real agent traces. There is no comparison with Nvidia hardware or other serving engines, no cost per token, and multi-node serving with prefill/decode disaggregation is only listed as future work. At 96k/32k the high-concurrency points take an hour or more each, so the ladder stops at 256 on TP8 and 768 on DP8+EP8.

For operators, the practical rule is simple. Divide the startup GPU KV cache size that vLLM prints by input plus output tokens; if expected concurrent sessions exceed that number, DP8+EP8 and possibly FP8 cache are worth testing. Otherwise TP8 gives the lower latency.