Nemotron 3.5 Super VL needs 16 H100s in BF16, AWS guide shows

Amazon Web Services has published a deployment guide for Nvidia’s Nemotron 3.5 Super VL on Amazon EKS, EKS Auto Mode and SageMaker HyperPod EKS. The key detail is hardware: in BF16 the model needs 16 H100 GPUs, or 8 in NVFP4.

What the AWS guide covers and what it needs

Nemotron 3.5 Super VL is a multimodal model with open weights, produced by distillation from the larger Nemotron 3 Ultra. According to AWS, it has 120 billion total parameters, of which 12 billion are active per token. It reads text, images, documents and video and outputs text, with a context length of up to 1 million tokens. Nvidia releases the weights together with the training data and post-training recipes, so teams can adapt the model to their own domain. It ships in BF16 and NVFP4 checkpoints.

The architecture mixes Mamba-2, mixture-of-experts and attention layers, with a RADIO vision encoder. AWS notes that sparse routing keeps the compute cost per token close to that of a far smaller model, but every weight still has to sit in GPU memory. That is why the minimum GPU counts matter. With vLLM, AWS lists these minimums for BF16 and NVFP4:

• B200 or GB200: 8 and 2
• B300 or GB300: 4 and 1
• H200: 8 and 4
• H100: 16 and 8

The reference implementation lives in the aws-do-eks project and covers cluster creation, weight download, deployment, testing, benchmarking and cleanup. The same Kubernetes manifests work on all three orchestrators. Each cluster uses CPU nodes for system pods and GPU nodes for the model, with Amazon ECR for images, a shared FSx for Lustre volume for weights and Elastic Fabric Adapter linking the GPU nodes. NVIDIA Dynamo serves the model either on one node or split into prefill and decode stages on separate nodes, and the image and video encoder can run on its own node too.

AWS says it validated the setup on p6-b200.48xlarge and p5en.48xlarge instances. The post does not mention pricing or regions.

How to size capacity from the benchmark sweeps

The most practical part of the post is a method rather than a result. AWS suggests running Nvidia’s aiperf tool at different concurrency levels, drawing latency limits as horizontal lines (its example: time to first token under 600 ms and inter-token latency under 100 ms), and reading off the highest concurrency that stays under both. In its two-node example that is c32, or about 320 tokens per second per replica. Dividing a target throughput by that figure gives the number of prefill-decode pairs to deploy.

AWS says the aggregated and disaggregated two-node setups showed about the same inter-token latency, with lower time to first token for the disaggregated one. The post shows this only as a chart, with no table of values.

The sweep also doubles as a diagnostic, according to AWS: if inter-token latency climbs, the fix is more decode capacity, while a climbing time to first token calls for more prefill capacity. That ties each symptom to a specific kind of GPU capacity.

Limits of a vendor guide for a model that is two weeks old

The model itself is new. Coverage from LetsDataScience says Nvidia released Nemotron 3.5 Super VL on October 15, alongside a 74B-parameter model called Bolt. That report cites Artificial Analysis for an Intelligence Index score of 26.5 for Super VL, and for a speed of about 630 output tokens per second at 100,000-token prompts, measured on a pre-release endpoint. Those are third-party figures from a different setup than AWS’s, so they cannot be compared with the 320 tokens per second per replica in the AWS example.

The benchmark scores in the AWS post, 63.44 percent on Hermes Bench, 87.05 percent on LangChain agent evaluations and 80.8 percent on CharXiv, are reported by Nvidia and are not independently verified in the post. AWS gives no comparison with other models.

The post also leaves open several questions for anyone budgeting a deployment: what 8 or 16 GPUs of these types cost per hour on EKS, how quality differs between BF16 and NVFP4, and how latency behaves with video inputs. The single-node and two-node results shown are for one topology, and AWS itself says results change with precision, parallelism shape and new Dynamo or vLLM releases.

For teams already running Kubernetes on AWS, the change is mainly one of effort: a tested path now exists to run an open vision-language agent model on infrastructure they control, instead of relying on a hosted API. For others, the GPU table is the first number to check. Related AWS agent work is covered in our report on Step Functions workflows callable by AI agents.