Vision-language-action models: How a robot turns words and camera images into movement

A vision-language-action (VLA) model is a single AI model that looks at camera images, reads an instruction in plain language, and outputs the movements a robot should make. It starts from a vision-language model, the kind that can describe pictures, and teaches it to “speak” robot actions as well as words.

The aim is a robot that follows everyday instructions without being programmed for each task.

A kitchen request, step by step

Imagine telling a robot arm: “Put the apple in the bowl.” A traditional system would need separate parts: one to find the apple, one to plan a path, one to control the motors. Each part is built and tuned by hand.

A VLA model collapses this into one loop. It receives the current camera image and your sentence. It outputs the next small motion, such as moving the gripper a few centimeters. Then it looks again and repeats, many times per second.

Teaching a language model to output movements

Language models predict the next piece of text, called a token. The trick behind most VLAs is to write robot actions in the same format. A gripper position, for example, can be turned into a short sequence of numbers that the model produces like words.

Google DeepMind’s RT-2 paper (2023) described this directly: the authors “express the actions as text tokens”, so actions share a format with language and can be mixed into the same training data.

Training then has two ingredients:

• Internet data, such as images paired with text, which gives the model general knowledge about objects and concepts.
• Robot demonstrations, which show what motions follow from what scenes and instructions.

The RT-2 authors reported that this mix helped robots handle new objects and commands absent from the robot data, such as placing an object on a particular number or icon. They also reported rudimentary reasoning, such as picking the smallest object. These results come from the authors’ own 6,000 evaluation trials.

From RT-2 to open models and smoother motion

RT-2 was submitted to arXiv in July 2023, with Anthony Brohan listed as first of 54 authors. It was followed by open and commercial work.

OpenVLA (2024) came from a group including Moo Jin Kim, Karl Pertsch, Sergey Levine, Percy Liang and Chelsea Finn. It is a 7-billion-parameter model built on a Llama 2 language model, with vision features from DINOv2 and SigLIP. It was trained on about 970,000 real-world robot demonstrations. The authors report it beat the 55-billion-parameter RT-2-X by 16.5 percentage points of task success across 29 tasks, with about seven times fewer parameters. They also say it can be fine-tuned on consumer GPUs using low-rank adaptation.

pi0 (2024, published at RSS 2025) was introduced by a team including Kevin Black, Karol Hausman, Sergey Levine and Chelsea Finn. The paper calls it a “vision-language-action flow model”. It uses flow matching, a method for generating smooth continuous values, on top of a pretrained vision-language model. The authors tested it on tasks such as laundry folding, table cleaning and box assembly across several robot types.

The shift matters because turning actions into word-like tokens can make motion coarse. Flow-based approaches aim for smoother control.

Where Google’s robot models fit in

Google DeepMind’s Gemini Robotics paper (March 2025) describes a VLA built on Gemini 2.0 that can directly control robots. The authors say it handles new object types and positions, unseen environments and open-ended instructions. With extra fine-tuning, they say it can learn new short tasks from as few as 100 demonstrations and adapt to new robot bodies.

The same paper describes a second model, Gemini Robotics-ER, which adds spatial and temporal understanding such as pointing, grasp prediction and 3D bounding boxes.

Our story on Gemini Robotics ER 2 in preview on Google Cloud shows how the pieces now fit together. Google describes ER 2 as a high-level planner that sits above a robot’s low-level controllers. Developers declare those controllers, such as VLA models, navigation APIs or custom functions, as tools. The planner decides what to do next, and the VLA carries out the physical step.

Think of a head chef and a skilled line cook. The planner says “pour until the glass is nearly full”. The VLA handles the hands.

Limits and open questions

VLA models are impressive in papers, but several limits apply.

• Data is scarce. Robot demonstrations cost far more than web text. Even OpenVLA’s roughly 970,000 demonstrations are small next to the data behind language models.
• Results are hard to compare. Labs test on different robots, tasks and success criteria. Figures such as the ones above are the authors’ own and may not carry over to your robot.
• Long tasks are hard. A VLA reacts to what it sees step by step. Multi-minute jobs need planning and checking, which is why a separate planner and a task verification step are often added.
• Safety and reliability. A wrong word in a chat is a nuisance. A wrong motion near people or fragile objects is not. The Gemini Robotics authors discuss safety considerations for this class of model.

What to watch

Three things are worth following. First, whether open models like OpenVLA close the gap with closed ones. Second, how well one VLA transfers across very different robot bodies, from arms to humanoids. Third, whether planner-plus-VLA setups, like the one Google is pushing through its cloud, prove more dependable than a single model trying to do everything.

Independent benchmarks on shared tasks would make these claims far easier to judge.