World Models: How AI Learns to Understand the Physical World
Artificial intelligence can identify objects, process language and discover complex patterns in large datasets. For machines operating in the physical world, however, pattern recognition alone is not enough. A robot should not only detect a box in front of it. It also needs to estimate how the box might move, whether it could fall and what would happen if it were pushed or lifted.
This is where world models become relevant. A world model gives an AI system an internal representation of its environment and the way that environment changes. It can help a machine predict possible outcomes, evaluate actions and plan before interacting with the real world.
What is a world model?
A world model is a learned representation of an environment and its dynamics. It receives observations such as images, sensor readings or system states and predicts how the observed situation may develop.
Depending on its purpose, a world model may learn:
which objects are present,
where those objects are located,
how they move,
how they interact,
how an action could change the situation.
There is no single architecture that defines all world models. Some predict future video frames or sensor readings. Others operate in a compressed latent representation and preserve only the information needed for planning or control.
The term became widely known in modern machine learning through the 2018 paper “World Models” by David Ha and Jürgen Schmidhuber. Their work demonstrated that an agent could learn a compact representation of a virtual environment and train a controller inside an internally generated version of that environment.
An internal simulation of possible futures
People regularly anticipate the consequences of their movements. Before picking up a glass, a person estimates its position, shape, stability and likely weight. Much of this happens before physical contact takes place.
A world model aims to give an artificial agent a comparable predictive capability. It may help answer questions such as:
What happens if the robot turns left?
Can the object be grasped safely from this angle?
Is a pedestrian likely to enter the vehicle’s path?
Which sequence of movements is most likely to complete the task?
The system can evaluate several possible futures internally rather than testing every option in reality. This is particularly valuable when real-world trials are expensive, slow, disruptive or dangerous.
How do world models work?
A practical world model usually combines several functions.
1. Perception
The system first receives information about its surroundings. Inputs may come from cameras, LiDAR, radar, ultrasonic sensors, force sensors, encoders or inertial measurement units.
2. State representation
Raw observations are converted into an internal state. This representation may encode objects, positions, movements, spatial relationships and other task-relevant properties.
The model does not necessarily need to retain every visual detail. For a warehouse robot, knowing that an obstacle blocks an aisle may be more important than reconstructing the obstacle’s exact colour and texture.
3. Dynamics prediction
The model learns how states evolve over time. It may predict that a moving object will continue along a trajectory, that opening a door creates a new path or that moving a robot arm changes the position of a held component.
4. Action conditioning
For an agent that can act, the model also needs to estimate how different actions affect future states. The prediction therefore depends not only on the current environment but also on the action being considered.
5. Planning
A planning or control system can compare predicted outcomes and select an action sequence that is likely to achieve the required objective.
Sensors provide the connection to reality
A world model depends on the quality and relevance of its observations. Sensors form the link between the physical environment and the AI system’s internal representation.
Cameras provide visual information. LiDAR generates spatial point clouds. Radar measures distance and relative velocity, while force and torque sensors capture physical contact. Inertial sensors estimate acceleration and orientation. Microphones, pressure sensors and environmental sensors may add further context.
The central challenge is often sensor fusion rather than the performance of one individual sensor. Measurements may be noisy, delayed, incomplete or contradictory. A robust model must combine them into a coherent estimate of the current situation.
Consider a pedestrian who temporarily disappears behind a parked vehicle. The person is no longer visible to a camera, but a capable world model should not assume that the pedestrian has ceased to exist. It should maintain an internal estimate and consider where the person may reappear.
This ability to represent objects beyond the current observation is essential for machines operating in dynamic environments.
Why world models matter for Physical AI
Physical AI refers to intelligent systems that perceive, decide and act in the real world. Examples include:
industrial robots,
autonomous vehicles,
warehouse robots,
drones,
humanoid robots,
intelligent production equipment,
service and assistance robots.
These machines need to understand spatial, temporal and physical relationships. Detecting an object is only the beginning. A robot may also need to determine how the object can be grasped, which forces are appropriate and whether its movement could endanger a person or damage nearby equipment.
Modern robotics research increasingly combines visual perception, language, spatial reasoning and action planning. Google DeepMind, for example, describes its Gemini Robotics systems in terms of capabilities including perception, spatial understanding, state estimation, planning and robot control.
Such systems are not necessarily complete world models in every technical sense. They nevertheless illustrate the transition from reactive object recognition towards context-aware machines that reason about the possible consequences of their actions.
Potential applications
Robotic manipulation
A robot can evaluate different grasping strategies and predict whether an object is likely to slip, collide with another item or become damaged. This is particularly useful when handling unfamiliar objects or materials.
Autonomous driving
An autonomous vehicle must interpret the current traffic situation and anticipate how other road users may move. A world model can compare possible scenarios and support safer trajectory planning.
Manufacturing and intralogistics
Mobile robots can account for the movements of workers, vehicles and goods. Production systems may use predictive models to adapt workflows when conditions change.
Drones and mobile systems
A drone can evaluate potential flight paths, avoid obstacles and account for moving objects or environmental influences.
Training in learned simulations
World models can function as data-driven simulators. Instead of training only in a manually programmed simulation, an agent can practise inside an environment whose dynamics have been learned from observations.
Generative world models extend this concept. DeepMind’s Genie research, for example, demonstrated how interactive and controllable virtual environments could be learned from video data.
The connection to generative video
Recent advances in generative video have brought world models to a wider audience. A video model must capture how scenes develop over time and how objects change their position, appearance and visibility.
A realistic video does not automatically demonstrate a reliable understanding of physics. A system may generate visually convincing sequences while still making fundamental errors involving causality, object permanence, geometry or physical interactions.
For an actionable world model, visual quality is not enough. The environment must respond consistently to actions, preserve relevant state over time and represent the consequences that matter for the agent’s task.
Systems such as Genie 3 are described as general-purpose world models capable of generating interactive environments that can be explored in real time.
World models at the edge
Many physical systems need to perceive, predict and react within milliseconds. Sending all sensor data to the cloud may introduce unacceptable latency, consume too much bandwidth or create privacy and availability risks.
This makes Edge AI an important part of the world-model discussion. A locally deployed model can support fast decisions and continue operating when connectivity is limited.
Running world models on embedded hardware remains difficult. These systems must operate with restricted computing power, memory and energy. Practical approaches may include:
smaller task-specific models,
compressed latent representations,
dedicated AI accelerators,
hybrid edge-cloud architectures,
event-driven processing,
different update rates for fast and slow processes.
Safety-critical applications also require predictable execution times. A highly accurate prediction is of limited value when it arrives too late for the machine to react.
Limitations and unresolved challenges
A world model is not a perfect copy of reality. It reflects the available training data, sensor inputs and assumptions built into the system.
Major challenges include:
Incomplete observations: Parts of the environment may be occluded or outside sensor range.
Rare situations: Unusual but important events may be poorly represented in training data.
Long-term prediction: Small errors can accumulate over many simulated steps.
Uncertainty: The future may contain several plausible outcomes. A model should represent this uncertainty rather than return a single overconfident prediction.
Generalisation: A model that performs well in one facility, city or weather condition may fail when the environment changes.
Computational cost: Detailed spatial and temporal predictions can require substantial resources.
Safety: A plausible prediction is not necessarily a safe one. Physical systems still require monitoring, operational limits and established safety mechanisms.
World model or digital twin?
World models and digital twins are related, but they are not the same concept.
A digital twin usually represents a specific machine, asset, production line or process. It may combine engineering models, design data, operating data and known physical relationships.
A world model is generally more strongly learned from observations. Its purpose is to capture environmental dynamics and predict future states or the consequences of actions.
The two approaches can complement one another. A digital twin may provide accurate domain knowledge, while a learned world model identifies patterns or interactions that are difficult to describe with manually designed equations.
From reactive systems to predictive intelligence
World models represent a major change in the way AI systems can be designed. Many conventional systems respond directly to an input. Machines operating in the physical world need an additional capability: they must track changes over time, consider alternative outcomes and plan their actions.
World models are therefore becoming an important building block for Physical AI, autonomous systems and advanced robotics. Their value does not depend on creating a perfect virtual copy of reality. The goal is to build an internal representation that is accurate enough for a machine to act more safely, efficiently and intelligently.
In brief
A world model is a learned internal representation of an environment. It enables an AI system to interpret its current state, predict how the situation may change and estimate the likely consequences of its actions.
