Flow matching: How robots and image models turn noise into smooth actions

Flow matching is a way to train a model that starts from random noise and gradually moves it into a useful result, such as an image or a robot’s next movements. The model learns a direction to move in at every point along the way. At run time, it follows those directions for a handful of small steps.

The idea is now used in image generators and in robot control. A recent robotics example is Ai2’s ManiFlow-tb, a model for a simulated robot arm that we covered earlier.

A robot arm that has to pick one of many good moves

Picture a robot arm above a table with three blocks. To stack the red block on the blue one, there is no single correct path. The arm can approach from the left or the right, high or low. All of these work.

A model that simply predicts the average of those paths can fail badly, because the average of "go left" and "go right" is "go straight into the block." A generative model avoids this. It learns the whole range of good moves and then picks one of them.

That is the job flow matching does for robots. It produces one plausible, consistent sequence of movements out of many possible ones.

Following arrows from noise to a finished answer

Think of a field of arrows laid over a map. Drop a marble anywhere and it drifts along the arrows until it lands on a spot that looks like real data. Random noise is the marble’s starting point. A finished image or action sequence is where it lands.

Training teaches a neural network to draw those arrows. During training, the model sees a real example and a random noise sample. It learns the direction that leads from the noise toward the example. In the simplest versions, that direction is a straight line.

At inference, the model starts from fresh noise. It asks the network for a direction, takes a small step, and asks again. Each round is called an integration step, because the steps add up to a path. Ai2’s model card for ManiFlow-tb lists 12 integration steps.

For robots, the network also receives context: camera images, the arm’s current state and the task. The noise is shaped like a block of future actions, and the result is the chunk the arm will carry out.

From diffusion models to straighter paths

Flow matching grew out of diffusion models, which generate images by removing noise bit by bit, often over dozens or hundreds of steps. Two 2022 papers pushed toward simpler training and shorter paths.

Yaron Lipman and colleagues at Meta published "Flow Matching for Generative Modeling" in October 2022. They described a simulation-free way to train continuous normalizing flows, a family of models that transform noise into data. They also treated diffusion paths as one special case and reported that optimal-transport-style paths speed up training and sampling.

In September 2022, Xingchao Liu, Chengyue Gong and Qiang Liu published rectified flow. It trains a model to follow straight paths between paired samples. The authors argue that straighter flows can be simulated accurately with coarse time steps, and they report good image results even with a single Euler step. The Euler step is the simplest way of moving along an arrow.

Both papers aim at the same practical result: fewer steps for similar quality. That matters most when a robot must act in real time.

Why robot models adopted it

Robot learning had already moved toward generative models. In 2023, Cheng Chi and colleagues published Diffusion Policy. It modeled a robot’s behavior as a denoising process and reported an average 46.9% improvement over earlier methods across 12 tasks from four benchmarks, partly because it handles several valid actions at once.

Flow matching offers a similar benefit with a simpler recipe. In October 2024, Physical Intelligence and collaborators released pi0, a vision-language-action flow model. The authors describe it as a flow matching architecture built on a pre-trained vision-language model. It was trained on data from several kinds of robots and tested on tasks such as laundry folding and table cleaning.

Ai2’s ManiFlow-tb belongs to the same family. According to its model card, it is a 322.5M-parameter action model that predicts 50 steps ahead and executes 25. It uses flow matching with consistency training and 12 integration steps. Consistency training is a method for teaching a model to jump along the path in fewer steps. The card gives no further detail on how it was applied.

Where flow matching falls short

Fewer steps do not mean free speed. Each step is a full pass through the network. A robot that runs 12 passes before every chunk of actions needs more computing than one that makes a single direct prediction.

Generated actions are also random by design. Ai2 reports that four evaluation runs of ManiFlow-tb on the same 128 held-out episodes gave 73, 75, 76 and 69 successes. The lab attributes the spread to simulation and batched inference that are not bitwise deterministic. Two runs agreed on only 82 to 97 of the 128 episodes. A model that samples its answers can succeed in one run and fail in the next.

The results also depend on the test setup. Ai2 says the time limit mattered as much as the model: with the default 30 seconds, only 26 to 28 episodes succeeded.

Finally, simulation results do not transfer automatically. ManiFlow-tb was trained on simulated Franka arm data, and the card does not claim real-world performance.

What to watch next

Three questions will decide how far flow matching goes in robotics.

• Fewer steps: methods such as consistency training aim to cut the 12 passes toward one or two, which would help on small onboard chips.
• Reproducibility: published seeds, fixed time limits and repeated runs make results comparable. The ManiFlow-tb spread shows why.
• Real robots: the test is whether models trained on simulated data hold up on physical arms.

Flow matching is a general tool, so the same ideas are likely to keep spreading across images, audio, video and motion. For robots, the practical question is simple: how few steps does it take to move well?