Nvidia’s surgical robot world model learns to predict video and actions

Nvidia has published Cosmos3-H-Embodied-Nano on Hugging Face, a 16-billion-parameter surgical robotics world model that predicts video, labels actions and proposes the next moves of a two-arm robot.

The model is a post-trained version of Nvidia’s Cosmos3-Nano. According to the model card, it ran for 50,000 iterations on the Open-H-Embodiment dataset, which spans 15 dataset families. It replaces Cosmos-H-Surgical-Simulator, which could only generate video from actions.

One model for three surgical tasks

The card describes three modes, trained together with mixing weights of 0.4 for forward dynamics, 0.2 for inverse dynamics and 0.4 for policy. In forward dynamics, the model takes a first frame plus a sequence of robot actions and produces video. In inverse dynamics it works backward, turning video into actions. In policy mode it receives a first frame, a text instruction and up to 16 past actions, and returns both actions and predicted video.

Each clip covers 2.4 seconds: 25 frames and 24 actions at 10 Hz. Output resolution is low, at 320×192 for 16:9 footage, 320×256 or 256×256. Actions are 20 values per step for a two-arm robot, covering each arm’s relative end-effector pose and a gripper command.

Under the hood, Nvidia uses a Mixture-of-Transformers design that pairs a Qwen3-VL-8B reasoning tower with a diffusion generation expert. The reasoning tower stays frozen, and post-training changes about 7B of the 16B parameters.

On a held-out validation set of 640 clips, Nvidia reports a video PSNR of 26.1 for forward dynamics and an action error (MSE) of 0.009 for inverse dynamics. For policy mode, the action error is 0.052 on average and 0.033 for the best of four samples, against 0.075 for a baseline that simply outputs the mean trajectory. Results are weaker on the CMR Versius system, at 0.101 average policy error and 18.6 PSNR, than on the dVRK group, at 0.046 and 26.9.

Who can run it today

The weights are released under the NVIDIA Open Model License Agreement, and the card says commercial and non-commercial use are both permitted. They run on Ampere, Blackwell and Hopper GPUs, were tested on H100, and only in BF16 on Linux. Inference goes through Nvidia’s cosmos-framework with a cosmos3_h_embodied plugin, and the card describes an HTTP policy server that returns 24 actions plus predicted video. The release date is listed as October 20, 2026.

The same day, Nvidia posted Cosmos3-H-Dreams, built on this model. It adds Versius action dimensions, is fine-tuned on cholecystectomy data and is distilled into a 4-step streaming student. Its card lists 30 frames per second at 832×480 on an RTX PRO 6000 Blackwell Workstation GPU with four steps, and 43.2 with two.

Gains over the predecessor come with trade-offs

The predecessor was built on the 2B Cosmos-Predict2.5 model and, according to its card, trained on about 26,500 demonstrations from 32 datasets. Its input was twelve 44-value action vectors at 512×288. The new model works at lower resolution but covers inverse dynamics and policy as well, which makes it usable for labeling recorded procedures and for testing a policy in simulation instead of on a patient or a physical rig.

Nvidia’s own comparison is mixed. On dVRK clips of 4.8 seconds, the new model scores a PSNR of 26.53 against 24.52. On 7.2-second Versius clips the two are roughly level on per-pixel metrics, and the predecessor wins on FVD (225 vs 274) and temporal consistency. Beyond about six seconds, the card says the new model’s frames blur more and its quality drops faster.

The training data is the Open-H-Embodiment collection, which Nvidia’s dataset page describes as 750 hours and 120,000 trajectories from 33 contributing organizations under CC-BY-4.0. The model card notes that 11 of its 15 families are public on Hugging Face, so the full training run cannot be reproduced from open data alone.

Limits the model card spells out

The card says the model is not a validated standalone system and recommends further testing before deployment. All figures come from Nvidia’s own validation split, and nothing in the source shows independent replication or any clinical evaluation. A mean action error says little about whether predicted instrument motion would be safe, and a 2.4-second window is short next to a real operation.

The data mix also leans heavily on a few sources. Johns Hopkins dVRK data makes up about 32% and CMR Versius about 20%, which may explain why Versius scores trail. Anyone planning to use the model for another robot would need to check how the 32 embodiment slots handle it.

For physical AI more broadly, the release shows Nvidia applying the same recipe used for general robotics to a regulated field: a large pretrained base, a frozen reasoning tower and a specialist action head. Its permissive license makes the weights a starting point for surgical robotics groups and companies that want simulation and synthetic data without building a world model from scratch.