AMD profiles DeepSeek-V4-Flash training: 556 ms to 239.8 ms per step

AMD has published a step-by-step profiling guide for training DeepSeek-V4-Flash in its Primus framework with madengine. On an eight-layer proxy, the unprofiled step falls from 556.0 ms to 239.8 ms, or 17,079 tokens/s per GPU.

One workload, five lenses, one changed field per run

The workload is a BF16 training proxy derived from the DeepSeek-V4-Flash Primus configuration, not the full model. AMD says it keeps representative V4 attention and mixture-of-experts operations. It has three sliding-window layers, three compressed-sparse layers and two heavily compressed layers, each with a 256-expert block and top-6 routing. It runs on one node with eight Instinct MI350X GPUs (the model tag says MI355X) in the rocm/primus:v26.7 image, with expert parallelism of 8, sequence length 4096 and global batch 8.

The method is to keep one model tag and change a single field in --additional-context for each madengine run. That field picks the collector: the PyTorch profiler for operators, rocm_trace_lite for kernel dispatch, or rocprofv3 for runtime kernels. TraceLens then reads each trace on the host through madengine report tracelens, which needs madengine v2.3.0 or later and writes the same CSV names whatever the source. The setup also needs a MAD repository checkout and enough disk: a rocprofv3 run writes roughly 150 MB of JSON per rank.

Fusions and MegaMoE deliver most of the speedup

AMD first runs eight unprofiled configurations, each changing one optimization family. Turning on Triton fusions for the small elementwise chains V4 adds (RMSNorm, RoPE, Sinkhorn and others) raises throughput 63.3%, from 7,367 to 12,028 tokens/s, and removes 215.5 ms per step. Grouped GEMM removes another 32.9 ms. Replacing both DeepEP and grouped GEMM with MegaMoE cuts a further 35.2 ms, and switching off recomputation saves 18.4 ms but lifts peak memory from 117.3 GB to 128.1 GB. Overall, the token rate rises 132%.

The profiler overhead is documented too. Measured against the 239.8 ms step, a rank-0 PyTorch window runs 1.09 times longer, an eight-rank window 1.20 times, rocm-trace-lite 1.44 times and rocprofv3 5.06 times. AMD therefore says to take durations from the unprofiled run and the rank-0 window, and to use the heavier collectors only to confirm kernel names.

The rank-0 window shows what is left to fix

In the 522 ms rank-0 window, computation takes 81.3% and exposed communication 4.3% (22.3 ms), while 14.3% (74.4 ms) is idle. MegaMoE stages account for 96.3 ms, or 22.6% of direct kernel time, and elementwise kernels for 71.0 ms. The Adam optimizer block at the end of each step takes about 23 ms.

The largest leftover target is the expert gradient: AMD attributes 71% of the aten::add_ time to autograd’s AccumulateGrad on the two grouped expert weights. Raising the global batch to 16 and 32 lifts throughput to 20,204 and 22,122 tokens/s per GPU, which AMD uses to show how much fixed cost shrinks with more microbatches.

A tooling play behind AMD’s DeepSeek push

The guide fits a pattern of AMD publishing practical software material around DeepSeek-V4. Last week it described reinforcement learning on 24 MI308X GPUs that lifted AIME to 49.2%, and on Nov. 4 it reported 1.72x more throughput from MI355X for coding agents. The commercial stakes are large: as we reported, AMD’s data center sales nearly doubled to $8.5B, with Q4 guidance above $16B.

Hardware sells better when customers can see where their own time goes. A reproducible recipe built on public images and open tooling lowers the cost of trying Instinct GPUs on a current mixture-of-experts model, which is likely the point of a post that teaches method rather than announcing a product.

What the numbers do not establish

AMD is explicit about the limits. Each rung is a single run, so there is no noise estimate or confidence interval, and the results apply only to this single-node, eight-layer proxy. They are not full-model DeepSeek-V4-Flash performance and not official AMD benchmarks. The setup also differs from real pretraining: it uses BF16 AdamW rather than the Muon optimizer DeepSeek used, mock data, forced uniform router load balancing and frozen indexer parameters. The 63.3% fusion gain and the MegaMoE gains should therefore be read as guidance on where to look, not as numbers to expect on a production cluster. Nothing here has been independently verified.