Research Preview

Introducing GWM Worlds 2

Interactive worlds generated in real time: continuous 720p video at 24 fps and audio at 48,000 Hz, responding to your inputs as you explore. Extending GWM Worlds, showcased in December last year, GWM Worlds 2 supports generated audio and rich subject and scene control.

September 3, 2026

by Runway

In December, we introduced GWM Worlds, a world model for real-time environment simulation. GWM Worlds extended our research efforts around real-time video generation, and excelled at maintaining spatial consistency across long sequences of movement.

Today, we're sharing GWM Worlds 2, the next iteration of our research to create interactive environments. Built on top of our foundational audio-video generation model, GWM Worlds 2 turns high-fidelity video and audio generation into real-time interactive simulation. You define the environment, subjects, visual style, physical rules and ambience. Once inside, you steer the world with text actions addressed to any subject or to the scene itself, alongside continuous camera motion. A text action can be a sword slash, a spoken reply, a room flooding with water, a dust storm rolling in or anything else you can put into words.

The world continues from each new input instead of following a fixed clip or script, so sessions have no preset length. This creates a foundation for interactive entertainment, virtual characters, robotics and embodied-agent simulation and generative design and interfaces.

WorldPrompt

Rich control over a world requires an input representation that generalizes across use cases. We base ours on the insight that a world can be split into two kinds of state: what persists and what changes over time. We formalize this as WorldPrompt, a format with two complementary layers.

• Persistent world context:
  • A genesis prompt that includes a scene description covering the environment, its layout, materials, lighting and ambient sound; the subjects that can participate in events, with their attributes; and the laws, reusable conventions that govern behavior, from gravity and collision to character abilities and the camera perspective.
  • A first frame to ground the generation visually.
• A timestamped event stream: Actions that describe movement, gestures, object interactions, speech and sound. Each action is a free-form text prompt with start and end timestamps, addressed to a subject or to the scene itself. Multiple actions can overlap. Camera input that represents a per-frame stream of viewpoint translation and rotation.

Example

Here is a concrete example from our dataset: a street-corner chat between two characters, written as WorldPrompt.

Finetuning the Model on WorldPrompt

We take our foundational audio-video generation model and finetune it on the WorldPrompt format. Below are some results showing the model follows the WorldPrompt structure correctly.

Real-Time Generation

The finetuned model is still too slow, and cannot yet generate autoregressively because it is still bidirectional. Therefore, we post-train the bidirectional model into a real-time autoregressive model, GWM Worlds 2. Unlike the bidirectional model, GWM Worlds 2 is also not restricted to a fixed duration and can generate indefinitely.

The same WorldPrompt structure applies here as well, but now the world responds as you play: each clip below was generated live at 24 fps, with a user steering the world through text actions addressed to subjects and the scene, plus continuous camera motion. Recurring actions can be bound to keys for fast play.

Model Overview

GWM Worlds 2 is an autoregressive diffusion video and audio model generating 720p video at 24 fps and audio at 48,000 Hz. GWM Worlds 2 conditions each AR step on three kinds of context: a global context (the genesis prompt and first frame), the current frame's inputs (camera input and the text actions that span the current frame) and the past generated frames, cached in a sliding window. The video and audio decoders are causal and run with a cache for faster decoding.

Using the Model

The model itself takes the WorldPrompt, with its persistent and event-based context, and outputs video and audio. There are various ways we could imagine using such a model:

• Ahead of time. The user (possibly with the assistance of an LLM) authors the timestamped event stream once at the start, and the model generates the whole video and audio with it. Uses: filmmaking, advertising.
• Turn-based. The generation runs until the user has to make a decision. The user then chooses an action, which an LLM could put into the timestamped event stream format. The model proceeds to generate until the next decision point. Uses: visual novels, interactive film.
• Real-time. The model continuously generates video and audio, and the user can take actions that immediately affect the generation stream. This is the most challenging scenario, as it requires text to be available with low latency, reacting immediately to the world. Making users type actions would be too slow, and even VLMs cannot react within tens of milliseconds. Uses: games, interactive experiences.

Demo

For some videos in the real-time generation section we used the ahead-of-time method to author rich actions beforehand without having to come up with them on the fly. For most of them we used our demo, which implements the real-time method.

Instead of coming up with text prompts on the fly, we bind key and mouse inputs to premade prompts. For example, the W key might be "The character moves forward," and left-click might be "The character throws a ball." While this works, we find that currently, the ahead-of-time method produces better quality, because the text prompts can describe much more of the scene. For example, if we turn around, we can describe the entire scene that becomes visible, whereas in the real-time demo the model has to come up with every detail on its own, with no anchor besides the persistent context and previous frames.

Continuing Play from a Video

Instead of starting from an image, we can also prefill the generation with an existing video. Here is an example where we prefill with a generated eight second video and then continue playing from it. Note that the model keeps the environment and audio consistent with the input video.

Agentic Control

We can let agents control the actions of the different subjects. Here is an example where we let an agent control both the character (to move and attack) and the environment (to direct its lighting). Thus, GWM Worlds 2 can be used as a simulated environment for evaluations of agents, and for training them in diverse environments.

We can also give the agents instructions of what they should achieve. As a simple test scenario, we generated a world with a humanoid robot and two flags: a red flag on the left and a blue flag on the right. The robot can only move forward, backward, left and right.

We instructed the agent to move to the red or the blue flag, where it succeeded on its first try in both cases. Then we made it first move to red then to blue, which it also succeeded at.

Multiplayer

We can also let different users control different subjects. This can be used to implement multiplayer experiences by letting each user control their own character, or by having one user control the world.

In the demo, we add this functionality as roles, where each role (e.g. player 1, player 2, director) can have its own set of actions addressed to its own subjects. The demo uses LiveKit to broadcast the video and audio to everyone accessing it, and each client can select a different role to participate in the experience.

World Authoring

In our demo, sessions are started from presets that contain the first frame, the genesis prompt and the possible actions for the different subjects, bound to different keys. Instead of manually typing out these prompts and coming up with actions, we allow users to create their own presets quickly by generating everything with the assistance of an LLM. A user can type a simple prompt like "third person perspective dirt bike in a snowy landscape" and the assistant takes care of generating everything. This way, we can go from an idea to a world within seconds.

Limitations

GWM Worlds 2 is a research preview, and real-time generation still trades fidelity for speed. Difficult camera inputs such as very quick rotations can cause details, textures and geometry to degrade. Long-term memory is also imperfect, and the model doesn't support image references beyond the first frame or prefilled video and audio.

While free-form text makes the control surface general, fully leveraging it can require an external real-time harness that tracks the state of the world and generates actions on the fly, for example the NPC dialogue in an NPC–player interaction.

Real-time video generation is still in its earliest stages, and the constraints outlined in this post will be solved with continued research. We expect the same curve that took offline video generation from short, rough clips to high-fidelity production footage to play out again here. GWM Worlds 1 and GWM Worlds 2 are two early points on that curve.

Fill out this form to get in touch about GWM Worlds 2

Full name*

Business email*

Company name*

Company size*

Select an option1-5051-200201-1,0001,001-10,00010,001+

Additional details

0/2000

Submit