Interactive world models: How AI generates a place you can steer as it unfolds

An interactive world model is an AI system that generates a video world one piece at a time, while you steer it. You type an instruction or move the camera, and the next piece of video shows the result. Nothing is stored in advance: the place is invented as you go.

Think of a game with no level files. Instead of loading a map someone built, a model draws each moment from what it has already shown you and what you just asked for. Turn left, and it has to decide what is there, then remember it when you turn back.

Drawing the next moment from the last one

An ordinary video generator takes a prompt and returns a finished clip. You wait, then watch. An interactive world model works in a loop instead. It makes a short stretch of video, accepts your next input, and makes the next stretch. This is called autoregressive generation: each step builds on the steps before it.

Two kinds of input steer it. Text can add an object, change the weather or describe an action. Camera controls move the viewpoint. The model combines both with a record of what already happened, so the new footage continues the old rather than starting over.

That record is the hard part. If the model forgets a building after you look away, the world falls apart. If it must reread every past frame at each step, it becomes too slow to use.

The memory trick that keeps a world in one piece

Most modern video generators are built on transformers, which use attention to relate every part of the input to every other part. During generation, a transformer can save intermediate results from earlier steps in a key/value (KV) cache, so it does not recompute them. For a world model, that cache works like a notebook of what has been shown.

Tencent’s Hy-Nomas, which we covered in our report on its release in the Hugging Face Diffusers library, uses this idea. According to Tencent, it generates a segment at a time and keeps a persistent KV cache that carries context from one segment to the next. Attention runs in one direction across segments, so a segment looks back at history but never forward.

There is a cost. Tencent’s model page says memory use grows with resolution, segment length and the amount of history kept. Longer memory means a bigger notebook, and a bigger notebook needs more GPU memory.

From a game dream to a worlds-on-demand race

The core idea is older than today’s video models. In 2018, David Ha and Jürgen Schmidhuber published a paper titled World Models. They trained a neural network to learn a compressed picture of a reinforcement learning environment, without labels. A small agent could then use that picture to act. The authors also showed that an agent could be trained entirely inside the model’s own generated simulation, then transferred back to the real environment.

In February 2024, a Google DeepMind team led by Jake Bruce, Michael Dennis, Ashley Edwards and Jack Parker-Holder published Genie: Generative Interactive Environments. The authors describe it as the first generative interactive environment trained without labels from Internet videos. The model has 11 billion parameters and three parts: a video tokenizer, a dynamics model that predicts the next frames, and a latent action model that infers actions from video. Users could act in the generated worlds frame by frame, even though no action labels were used in training.

DeepMind announced Genie 3 on August 5, 2025. It says the model generates explorable environments from a text prompt at 720p and 24 frames per second, in real time. Tencent’s Hy-Nomas now targets the same space under an Apache-2.0 license.

Who is building them, and for what

Two uses stand out. The first is entertainment and creative tools: explorable scenes, game-like prototypes and video you direct while it plays. The second is training AI agents. DeepMind says it generated worlds for a recent version of its SIMA agent, a generalist agent for 3D environments. SIMA pursued goals by sending navigation actions, and Genie 3 simulated the results without knowing the goals.

Hy-Nomas follows a different path from Genie 3 in how it is shared. Tencent published a BF16 version of the distilled model for Diffusers. Its example uses 4 denoising steps per segment at 704 x 1280 and 24 frames per second. The model has about 9 billion parameters and runs on one CUDA GPU in that example.

Tencent’s main repository describes a standard setup with 8-way sequence parallelism, plus a separate RTX 4090 setup that uses 8-bit weights and cache and CPU offload. The Diffusers page says the install currently comes from a development branch, not a released version.

Where the illusion breaks

World models are impressive, but they are not physics engines. They predict what footage should look like, and they can get it wrong. DeepMind lists several limits for Genie 3: agents have a limited set of direct actions, interaction between multiple independent agents is an open research problem, real-world places are not rendered with perfect accuracy, and readable text usually appears only when it is in the prompt. It adds that interaction lasts a few minutes, not hours, and that visual memory reaches back about a minute.

Drift is the common failure. Small errors in one segment become the starting point for the next, so a world can slowly change in ways nobody asked for. How long a world stays coherent is hard to measure, and published figures are mostly vendor statements.

That caution applies to Hy-Nomas too. Tencent’s Diffusers page gives no quality scores and no measure of how many segments can be chained before drift. Tencent released Hy-Nomas-Bench, four benchmarks with 280 test cases covering text interaction, camera navigation, mixed control and spatial memory, but the Diffusers page reports no scores from them. The main repository lists the technical report as coming soon.

What to watch next

Three things will show whether this field is maturing. The first is memory length: how many minutes a model can hold a place steady, as measured by independent tests rather than the builders. The second is cost. The fact that Tencent offers a one-GPU path, and a separate consumer-card configuration, points toward lower hardware needs, though the Diffusers page gives no speed or memory figures.

The third is openness. A model that people can download, run and inspect can be tested by others, and that is how the drift question gets answered. Open releases, including Hy-Nomas, make that kind of comparison possible, though Tencent has not yet compared it with Genie 3 or other world models.