World Models: The AI That Promises to Simulate Everything, or at Least Make Some Very Cool Videos
World models are the new AI frontier, promising to simulate the physical world - or at least generate really cool videos - while attracting billions in funding and a lot of fuzzy definitions.
Over the past few years, we've all gotten a crash course in what companies call artificial intelligence - which has mostly meant large language models (LLMs). But LLMs are no longer the only AI category attracting massive funding and breathless hype. Enter "world models," which aim to simulate the physical world, or at least a useful approximation of it. Instead of just chatting, these systems want to let you poke around a 3D environment like it's a video game, but with more math and fewer loot boxes.
To understand what's different about world models, Ars spoke with three experts: Vincent Sitzmann from MIT, Anastasis Germanidis from Runway, and Ben Mildenhall from World Labs. Their consensus: while LLMs started with a chat interface and then looked for a use case, world models are starting with specific applications - robotics, research, asset generation - and haven't quite figured out what the interface should look like. Think of it as AI working backward, which is on brand.
Not everyone is sold on LLMs as the path to general intelligence. Former Meta chief AI scientist Yann LeCun told Wired that the idea of LLMs achieving human-level intelligence is "complete nonsense." Fei-Fei Li, computer vision pioneer and co-founder of World Labs, wrote that LLMs are "wordsmiths in the dark; eloquent but inexperienced, knowledgeable but ungrounded." Even Hugging Face CEO Clem Delangue thinks we're in an "LLM bubble" that might burst next year. So world models are the new hotness, at least until something even hotter comes along.
World models have already attracted eye-popping funding: World Labs and AMI reportedly raised around $1 billion each in February and March, and Runway raised $315 million in February. That's a lot of money for a term that, as Sitzmann admits, is "definitely overloaded." His definition: any model that takes an interaction and simulates what happens next. Runway's definition adds that it builds an internal representation of an environment. Mildenhall says the key distinction from LLMs is continuous, real-time interaction - not the turn-based text exchange of "A then B then A then B." Fei-Fei Li offers three criteria: perceptual, geometrical, and physical consistency; multimodality; and the ability to output next states based on input actions.
But "world model" is also becoming a branding term. Companies slap it on everything from action-conditioned video generation to 3D asset creation to robot policy evaluation. For now, says Sitzmann, "what most people mean when they say 'world model' is generating pixels" - i.e., realistic video conditioned on actions. That's a natural extension of video generation models, which Runway already does. As Germanidis put it, when they added camera controls to their video tools, "it started to feel more almost like a video game."
The technical challenge: most video diffusion models generate all frames at once, denoising the whole sequence in one go. That's great for coherence but terrible for interactivity - you can't intervene mid-generation. The solution for world models is autoregressive diffusion, which denoises frames sequentially. Users can provide actions that influence the next frames in real time. The downside: it's compute-expensive, and the simulation tends to drift, forgetting earlier details. Germanidis hopes that 3D consistency will "emerge from scale," a nod to what computer scientist Richard Sutton called "the bitter lesson": explicit human knowledge often gets in the way. So they're betting big on brute force.
In short, world models are where AI goes to simulate reality, one expensive denoising step at a time. Whether they'll deliver on the promise or just produce very fancy deepfakes remains to be seen - but the funding rounds suggest investors are willing to find out.
The Good Times
News in your inbox.
One sardonic roundup, delivered on your schedule. Free. Unsubscribe whenever your tolerance for wit runs out.
Already subscribed but we never reach your inbox? Check your spam folder and hit 'Not spam' (or 'Remove from spam') to bust us out of junk-mail purgatory. You'll be helping everyone else too.
Rewrite Article
Select parts to regenerate with a fresh AI pass. Translations will be updated automatically.
Generate AI Image
Creates a sardonic version of the article image using OpenAI.