Praxis-1: How Runway Turned a Video Foundation Model Into a Robot Brain

Praxis-1: How Runway Turned a Video Foundation Model Into a Robot Brain Runway is best known for video generation tools used by filmmakers and designers. But the company just released something that points in a very different direction: Praxis-1, a 27-billion-parameter open-weight model that controls real robots by predicting what the world will look like next — and what actions will get it there. The model is now available under the Apache 2.0 license, and it currently ranks first among world action models on NVIDIA's RoboLab-120 leaderboard, outperforming alternatives like FLUX 3 Action, Cosmos 3 Nano, and π0.5. What makes it interesting isn't just the benchmark number — it's the architectural bet underneath it. The Core Idea: Video Pretraining as a Shortcut to Robot Control Training a robot policy from scratch requires enormous amounts of robot-specific data, which is expensive and slow to collect. Praxis-1 sidesteps much of that cost by starting from a model that already understands how the physical world works — Runway's own video foundation model, trained on general video across twelve generation tasks. The intuition is straightforward: a model that can predict what a scene will look like after a hand reaches for a cup has already learned something about object permanence, contact dynamics, and spatial relationships. That knowledge transfers. The robot-specific fine-tuning then becomes a much smaller lift. This is the "world action model" framing: rather than mapping observations directly to motor commands, Praxis-1 first imagines what the world will look like after the next action, then reads that imagined future to decide what motion to output. Architecture: Two Models Working in Tandem Praxis-1's architecture has two main components that run together at inference time. The video backbone is a 26.9B-parameter mixture-of-experts diffusion transformer built on Runway's video foundation model. It uses a Wan2.2 VAE for latent encoding and a frozen Qwen3.5-2B text encoder for instructions. Of its 20 feed-forward layers, 17 are MoE layers — each with one shared expert and 128 routed experts, of which each token activates eight. This keeps active compute roughly equivalent to a dense model while holding far more total parameters. The backbone predicts future video in 1.6-second chunks of six latent frames. The action expert is a much smaller dense 16-layer transformer (~314M parameters) that reads the video backbone's internal state as it imagines the future. It alternates between cross-attention into eight spread-out backbone encoder blocks and self-attention over its own tokens. The key design choice: the action expert reads the backbone's predicted future frames, not just the current observation. It learns to act based on what the model thinks will happen, not just what it sees now. At inference, the robot sends camera frames and its current pose, receives 24 action rows (1.6 seconds at 15 Hz) alongside predicted future video latents, executes the chunk, and calls again. The round-trip on a single H100 takes roughly 0.55–0.6 seconds. Four Training Stages The training pipeline is staged carefully to avoid wasting robot data on a model that hasn't yet learned basic physics: Video pretraining — The backbone starts as Runway's general video model, trained on clips up to 20 seconds with full-sequence attention. Embodied-video mid-training — The backbone switches to chunk-causal attention and trains on a mix of egocentric human video (58%), general video (20%), and real robot footage (18%). No action targets yet. Video and action joint mid-training — The action expert is attached with fresh weights. The loss is an equal-weight sum of video and action velocity losses. The Muon optimizer handles most weight matrices; AdamW handles embeddings and projections. Embodiment-specific fine-tuning — The full mid-trained model is fine-tuned for a specific robot. For the DROID Franka arm, this used ~277 hours of successful episodes across 50k optimizer steps on 128 H100s. More mid-training compute consistently improves downstream fine-tuning. On the unseen RT-1 embodiment, SimplerEnv success rose from 56.3% at the earliest checkpoint to 80.0% at the latest — without any RT-1 data seen during mid-training. Results That Hold Up on Real Hardware On NVIDIA's RoboLab-120 benchmark (120 tabletop tasks, 10 trials each), Praxis-1 fine-tuned on DROID solved 555 of 1,200 episodes (46.2%), placing first among world action models and open-source robot foundation models. The next-best world action model, FLUX 3 Action, scored 42.9%. On a real Franka FR3 station running FLUX 3 Action's published ten tasks, Praxis-1 achieved 88.0% success versus FLUX 3 Action's 80.0%. The model also adapts quickly to new embodiments. Fine-tuned on just 29 minutes of teleoperated data from Noble Machines' robot, it succeeded in all 15 simulated trials across two trained layouts. Fine-tuned on Google's RT-1 dataset, it reached 80.0% on SimplerEnv — well above RT-1's own 52.4%. One offline analysis is worth noting: when the team measured action error on held-out DROID windows, most of the error came from which future the video model imagined, not from the action expert's ability to read that future. The bottleneck is video prediction quality, not action decoding. Quantization and Deployment Praxis-1 supports 4-bit NVFP4 quantization for its routed MoE experts. On RoboLab-120, the quantized model scored 46.5% versus 47.8% in bf16 — not statistically significant. Weights drop from 54.4 GB to 17.5 GB, making deployment on a single 80 GB GPU practical. Real-time chunking with streaming optimization brings server time down from ~545 ms to 128 ms for bimanual tasks. What This Means for Practitioners The most direct implication is that general video pretraining is a credible path to robot control. If you have a new robot embodiment, Praxis-1's mid-trained base model gives you a starting point that already understands physical dynamics, requiring far less fine-tuning data than training from scratch. Three checkpoints are available on Hugging Face under Apache 2.0: Praxis-1 fine-tuned on DROID, fine-tuned on bimanual YAM, and the base mid-trained model. The release includes policy servers, MuJoCo simulation environments, and native LeRobot support for LoRA or full fine-tuning. The cost comparison to a frontier LLM agent is telling: on the same bimanual YAM station, GPT-6 Astra took 15.6x longer to finish a task and cost 51x more per completed task than Praxis-1. For real-world deployments where latency and cost matter, a specialized world action model has a clear edge. Limitations Worth Knowing Praxis-1 struggles with look-alike objects — it can reach for the wrong item when two objects are visually similar. Vague instructions that omit specific attributes (color, size) also degrade performance. Long multi-step tasks from a single prompt remain challenging without step-by-step breakdown. These are honest limitations that the team documents clearly in the technical report. Conclusion Praxis-1 is a concrete demonstration that video pretraining at scale transfers to robot control in ways that matter for real deployments. The architecture — a large video backbone imagining the future, a small action expert reading that imagination — is clean and well-motivated. The open-weight release under Apache 2.0 makes it immediately usable for researchers and practitioners working on robot learning. Whether this approach scales further, and whether the video prediction bottleneck can be closed, are open questions. But as a starting point for embodied AI work, Praxis-1 is worth a close look.