GLM-5.3 needs two DGX Stations to run locally, and peaks at 607.8 tokens a second

Nvidia has published a developer tutorial for serving GLM-5.3 NVFP4 across two DGX Stations with vLLM. The checkpoint is about 464 GB, more than the 252 GB of HBM3e on one GB300 GPU. Nvidia’s benchmark peaks at 607.8 tokens per second.

Two machines, one OpenAI-compatible endpoint

The model is Nvidia’s quantized version of Z.ai’s mixture-of-experts GLM-5.3. It has 753 billion parameters, with 40 billion active per token. Nvidia notes that sparse activation cuts compute, but every weight must stay available because each token can be routed to different experts.

The setup uses tensor parallelism with TP=2. Each GB300 GPU holds and computes roughly half the model, and the two Stations exchange partial results over a ConnectX-8 RDMA link. Station A hosts the API and Station B runs only a worker. Nvidia reports about 218 GiB of weights per GPU, and total GPU memory in use of about 231 GiB on each Station once loaded.

The guide requires two GB300 DGX Stations running Linux, connected and validated with Nvidia’s separate Connect Two DGX Stations playbook. Each Station needs at least 464 GB of free disk for its own copy of the checkpoint. Users must accept the NVIDIA Open Model Agreement and the original GLM-5.3 license terms. The benchmark step also needs a Hugging Face token with access to the gated cais/hle dataset.

The launch script pins a vLLM container image and uses multi-token prediction with four draft tokens per step, an FP8 KV cache and prefix caching. It sets a context ceiling of --max-model-len 98304 tokens and --gpu-memory-utilization 0.93. Tool calling is enabled, so agent frameworks that use the OpenAI tools API can call the same endpoint.

Throughput peaks at 16 requests, then collapses

Nvidia measured the setup with its AIPerf tool on the 80 coding conversations in SPEED-Bench, using the model’s default maximum reasoning effort and an 8,192-token output cap. Prefix caching was on.

• Concurrency 1: 107.4 tokens per second overall, 49.84 s mean request latency
• Concurrency 8: 386.0 tokens per second, 114.17 s
• Concurrency 16: 607.8 tokens per second, 136.46 s
• Concurrency 32: 385.0 tokens per second, 453.54 s, with 77.90 ms between tokens

Per-user speed also drops as load rises, from 115.3 tokens per second at one request to 13.0 at 32. Time to first token stays under 500 ms up to concurrency 16, then reaches 899.36 ms at 32. Nvidia says the concurrency 32 runs need --max-num-seqs 32, while the published launch script caps it at 16, so the two settings must be changed together when reproducing the table.

What the guide leaves unanswered

The numbers are Nvidia’s own, from one benchmark on one coding dataset, and the post does not say whether anyone outside the company has reproduced them. Throughput will also differ with other prompts, because the speedup from multi-token prediction depends on how often the drafts are accepted. Nvidia says as much, and tells readers to rerun AIPerf with their own prompts.

Several points are missing. The post gives no price for two DGX Stations and no availability by region. It also does not compare the setup with hosted endpoints on cost or speed, even though it pitches local use for sensitive or proprietary data and for prototyping without per-token costs. Nor does it measure output quality after quantization or the effect of the FP8 KV cache. The only check shown is a short test prompt. The security note is blunt: the API is unauthenticated and exposed on every Station A interface, so the port must be firewalled and given authentication and TLS before wider use.

Concurrency is the practical ceiling. A single multi-turn agent may sit near the 107.4 tokens per second single-request figure, while a small team sharing the endpoint would land in the 8 to 16 range. Past that, the table shows latency growing faster than throughput.

GLM-5.3 is also appearing on other hardware. Our earlier coverage of AMD’s GLM-5.3-Flash work on MI300X reported that switching the parallelism strategy doubled long-context throughput. We also covered AMD’s claim that GLM-5.2 on eight MI355X GPUs reads 1M tokens 7x faster. Those cases involve different model versions and larger accelerator counts, so they are not like-for-like with Nvidia’s two-Station figures.

Nvidia closes the post with the line “A frontier-scale model is now ready to build on, right from your desk.” The follow-up it points to is the Local AI recipe repository, which it says holds the latest recommended configurations.