Latent video diffusion: How AI models turn noise into video, frame by frame

A latent video diffusion model makes a video by starting with random static and removing the noise in many small steps, until a clip that matches a text prompt remains. It does this on a compressed version of the video, not on raw pixels, which makes the job affordable.

This family of techniques sits behind many of today’s AI video tools. Grok Imagine’s seven-minute Odyssey episode is a recent public example, though the company has not published how that piece was made.

Cleaning up static the way a restorer cleans a painting

Picture a photograph buried under heavy grain. A skilled restorer does not repaint it in one go. They remove a little grain, step back, look at what the picture seems to be, and remove a little more.

A diffusion model works like that restorer. During training, it sees real images or clips that have been covered in noise by varying amounts, and it learns to predict the noise so it can be taken away. At generation time it begins with pure noise and applies that skill repeatedly. A text prompt steers each step, so the clean-up drifts toward “a ship on a stormy sea” and not just any plausible picture.

Working in a compressed space to save compute

Video is enormous. A single minute holds well over a thousand frames, each with millions of pixels. Denoising all of that directly would be very slow.

The fix is called a latent space. A separate network, an autoencoder, squeezes each frame into a much smaller set of numbers that keeps the important content and drops fine pixel detail. The diffusion model works only on these compact numbers. When it finishes, the autoencoder’s decoder expands the result back into full-resolution frames.

Think of it as sketching on a small notepad and then enlarging the sketch, instead of painting at full wall size from the start.

Adding time so frames agree with each other

An image model can make one good frame. A video needs hundreds that agree with each other: the same face, the same lighting, an object that moves smoothly instead of flickering.

In 2023, Andreas Blattmann, Robin Rombach and colleagues at Stability AI, LMU Munich, NVIDIA and other institutions described one approach in “Align your Latents” (CVPR 2023). They first trained an image latent diffusion model, then added a time dimension to its latent space and fine-tuned it on encoded video sequences. Only the new temporal layers needed training when building on the public Stable Diffusion model, according to the paper.

Those temporal layers let each frame “look at” its neighbors while it is being denoised. That is what keeps a character’s beard or a wave’s shape steady from one moment to the next.

Where the idea came from, and how it scaled

The pieces arrived one at a time:

• 2020: Jonathan Ho, Ajay Jain and Pieter Abbeel published “Denoising Diffusion Probabilistic Models”, showing that this step-by-step denoising could produce high-quality images.
• 2021: Rombach, Blattmann, Dorenz, Esser and Ommer introduced latent diffusion models, moving the process into an autoencoder’s compressed space to cut training and inference cost. This is the basis of Stable Diffusion.
• 2022: William Peebles and Saining Xie proposed the Diffusion Transformer (DiT), replacing the usual U-Net backbone with a transformer working on patches of the latent image. They reported that more compute consistently gave better image quality.
• 2023: Stability AI’s Stable Video Diffusion paper described three training stages: text-to-image pretraining, video pretraining and high-quality video fine-tuning. It argued that a carefully curated dataset is necessary for good video, and released code and weights.
• 2024: OpenAI’s Sora report, dated February 15, described a transformer working on “spacetime patches” of compressed video, with a compression network that shrinks video in both space and time. OpenAI left out model and implementation details.

The trend is plain: compress first, then let a transformer, which scales well with more data and compute, do the denoising.

Convincing shots, weak staying power

This design is good at making individual moments look real. It is much less proven at telling a story. TechCrunch’s Anthony Ha, reviewing the Grok Imagine episode, wrote that some moments could pass for real actors on a real sea, yet the whole felt airless, with no momentum. That is one reviewer’s opinion, but it matches a known structural limit.

Several limits follow from how these models work:

• Short memory. A model denoises a limited stretch of footage at a time. Keeping a character, costume or location identical across minutes usually needs extra tricks, such as feeding in reference images or stitching clips together. Which tricks any given product uses is rarely disclosed.
• Detail lost in compression. The autoencoder discards fine texture, which is why small text, hands and fast motion can look off.
• No built-in physics or script. The model learns what footage tends to look like. It does not hold a plot or a rulebook for how objects behave.
• Cost. Each clip needs many denoising steps, and longer or higher-resolution video multiplies the work.

Human direction fills those gaps. The Grok post credits a collaboration with Wonder Studios and director Dev Walker, but, per our reporting, how much of the work was done by people rather than the model is unknown.

What to watch as clips grow into episodes

Three things are worth following. First, whether makers explain their process: the Grok account asked viewers whether they wanted a full breakdown of how the episode was made, which would be a rare look at the human steps involved. Second, whether models can hold characters and scenes steady over minutes without manual stitching. Third, how well sound and dialogue are generated alongside the image, since stilted lines were a main complaint in the review.

Until those improve, expect latent video diffusion to be strongest at short, striking shots and weakest at the long, patient work of keeping an audience watching.