Research Preview
Introducing GWM Worlds 2
Interactive worlds generated in real time: continuous 720p video at 24 fps and audio at 48000 Hz, responding to your inputs as you explore.
September 3, 2026
by Runway
Built on top of our foundational audio-visual generation model, GWM Worlds 2 turns high-fidelity video and audio generation into real-time interactive simulation. You define the environment, subjects, visual style, physical rules and ambience. Once inside, you steer the world with text actions addressed to any subject or to the scene itself, alongside continuous camera motion. A text action can carry a movement, a gesture, a line of dialogue or a change in the weather.
The world continues from each new input instead of following a fixed clip or script so sessions have no preset length. This creates a foundation for interactive entertainment, virtual characters, robotics and embodied-agent simulation, and generative design and interfaces. Extending GWM Worlds, showcased in December last year, GWM Worlds 2 supports generated audio and rich subject and scene control.
Playing the survivor. A desert world played in first person: the survivor moves, jabs the spear, drinks from a canteen and jumps.
Directing the scene. The same world from the director's seat: prompts addressed to the weather and the campfire turn sunset into a starry night, flare the fire and summon a dust storm.
Both at once. A player crosses the desert as the survivor — walking, thrusting the spear, jumping — while a director reshapes the scene around them: the campfire flares and the sunset fades into a starry night.
Capabilities
The clips below were generated with GWM Worlds 2. In each one, a user is steering the world with text actions addressed to subjects and the scene, plus continuous camera motion. Recurring actions can be bound to keys for fast play.
The model supports both first-person and third-person navigation like walking, driving, riding, looking around and flying through a world. The camera and the subject can be controlled independently.
Subjects and the scene can be directed to take arbitrary actions: movement like running, climbing and jetpack bursts; object interactions like firing a laser pistol, switching a lamp on or opening an umbrella; and scene events like flooding a room.
Characters can hold conversations and control the voice, tone and language of the speech, e.g. NPC–player dialogue or a vlogger talking to the camera. Lip movement and delivery are generated to match.
WorldPrompt
Rich control over a world requires an input representation that generalizes across use cases. We base ours on the insight that a world can be split into two kinds of state: what persists and what changes over time. We formalize this as WorldPrompt, a format with two complementary layers.
- Persistent world context
- A genesis prompt that includes a scene description covering the environment, its layout, materials, lighting and ambient sound; the subjects that can participate in events, with their attributes; and the laws, reusable conventions that govern behavior, from gravity and collision to character abilities and the camera perspective.
- A first frame to ground the generation visually.
- A timestamped event stream
- Actions that describe movement, gestures, object interactions, speech and sound. Each action is a free-form text prompt with start and end timestamps, addressed to a subject or to the scene itself. Multiple actions can overlap.
- Camera input to represent a per-frame stream of viewpoint translation and rotation.
Example
Here is a concrete example: a street-corner chat between two characters, written as WorldPrompt.
Scene
An urban wet pavement lined with brick buildings, stone trim, and potted round bushes on the right. A wet asphalt road with yellow lines runs alongside the pavement. Reflections are visible on the wet ground, scattered with yellow autumn leaves.
Subjects
The street sweeper. A slender male street sweeper with dark short hair, wearing an orange high-visibility vest over a grey sweatshirt, dark trousers, and dark boots. He holds a broom and a dustpan shovel, and speaks with a male voice. Located on the wet pavement to the left.
The woman in green trench coat. A slender woman with dark hair tied in a ponytail, wearing a green trench coat, dark pants, and black boots. She speaks with a female voice and walks down the wet pavement.
Law
Gravity behaves like Earth, causing leaves to lie flat on the wet pavement. The camera follows the woman in green trench coat in third-person view.
Move over or focus a timeline bar to see its full prompt.
Model Overview
GWM Worlds 2 is an autoregressive diffusion video and audio model generating 720p video at 24 fps and audio at 48000 Hz. GWM Worlds 2 conditions each AR step on three kinds of context: a global context (the genesis prompt and first frame), the current frame's inputs (camera input and the text actions that span the current frame) and the past generated frames, cached in a sliding window. The video and audio decoders are causal and run with a cache for faster decoding.
Attention pattern. Every token attends to the global tokens (first frame and genesis prompt), and each frame's video, text and audio tokens causally attend to themselves and past frames in a sliding KV-cache window; older frames are evicted. Global tokens attend only themselves. Per frame: v = video, a = audio, t = text, c = camera.
Using the Model
The model itself takes the WorldPrompt, with its persistent and event-based context, and outputs video and audio. There are various ways we could imagine using such a model:
- Ahead of time. The user (possibly with the assistance of an LLM) authors the timestamped event stream once at the start, and the model generates the whole video and audio with it. Uses: filmmaking, directing.
- Turn-based. The generation runs until the user has to make a decision. The user then chooses an action, which an LLM could put into the timestamped event stream format. The model proceeds to generate until the next decision point. Uses: visual novels.
- Real-time. The model continuously generates video and audio, and the user can take actions that immediately affect the generation stream. This is the most challenging scenario, as it requires text to be available with low latency, reacting immediately to the world. Making users type actions would be too slow, and even VLMs cannot react within tens of milliseconds. Uses: games, interactive experiences.
Demo
For some videos in the capabilities section we used the ahead-of-time method to author rich actions beforehand without having to come up with them on the fly. For most of them we used our demo, which implements the real-time method.
Instead of coming up with text prompts on the fly, we bind key and mouse inputs to premade prompts. For example, the W key might be "The character moves forward", and left-click might be "The character throws a ball". While this works, we found the ahead-of-time method produces better quality because the text prompts can describe much more of the scene. For example, if we turn around, we can describe the entire scene that becomes visible, whereas in the real-time demo the model has to come up with every detail on its own, with no anchor besides the persistent context and previous frames.
The real-time demo. A third-person dirt bike in a snowy landscape, played live: key and mouse bindings fire premade action prompts while mouse movement steers the camera.
Multiplayer
Because the model knows the concept of subjects, we can also let different users control different subjects. This can be used to implement multiplayer experiences by letting each user control their own character or by having one user control the world.
In the demo, we add this functionality as roles, where each role (e.g. player 1, player 2, director) can have its own set of actions addressed to its own subjects. The demo uses LiveKit to broadcast the video and audio to everyone accessing it, and each client can select a different role to participate in the experience.
World Authoring
In our demo, sessions are started from presets that contain the first frame, the genesis prompt and the possible actions for the different subjects, bound to different keys. Instead of manually typing out these prompts and coming up with actions, we allow users to create their own presets quickly by generating everything with the assistance of an LLM. A user can type a simple prompt like "third person perspective dirt bike in a snowy landscape" and the assistant takes care of generating everything. This way, we can go from an idea to a world within seconds.
1. Describe the world you want
2. A first frame is generated
3. The genesis prompt is drafted
4. Subject actions are bound to keys
From prompt to preset. The authoring flow, step by step: describe the world, get a generated first frame, a drafted genesis prompt and a set of subject actions bound to keys — all editable before you hit play.
Limitations
GWM Worlds 2 is a research preview, and real-time generation still trades fidelity for speed. Over time, details, textures and geometry can drift, especially with quick camera movements. Long-term consistency is also imperfect, and the model doesn't support image references beyond the first frame.
While free-form text makes the control surface general, fully leveraging it can require an external real-time harness that tracks the state of the world and generates actions on the fly, for example the NPC dialogue in an NPC–player interaction.
Real-time video generation is still in its earliest stages, and the constraints outlined in this post will be solved with continued research. We expect the same curve that took offline video generation from short, rough clips to high-fidelity production footage to play out again here. GWM Worlds 2 is a very early point on that curve.