At RSS 2026, Georgia Tech assistant professor and NVIDIA researcher Danfei Xu delivered a sharp critique of the prevailing belief that high-fidelity video generation equates to robot planning. In his talk "Compositional World Models," Xu argued that the ability to generate a visually perfect future video does not mean the robot can physically execute the corresponding actions—a gap he calls the "video-action gap."
Xu's core insight is that generation and planning are fundamentally different. Generation samples high-probability outcomes from a data distribution, while planning must construct valid solutions for novel, long-horizon tasks under physical constraints—solutions that may not exist in the training data. For example, a robot trained on separate skills like "carry heavy object," "open fridge," and "place on shelf" may fail when asked to combine them in a new sequence, such as trying to open the fridge while holding the object.
To bridge this gap, Xu proposes compositional world models inspired by Marvin Minsky's modularity. The key idea is to decompose complex tasks into reusable skill modules that can be dynamically composed at test time. Instead of linear chaining, which suffers from inefficient information propagation, Xu uses factor graphs to define spatial and temporal constraints across skills. This allows, for instance, coordinating two robot arms without dedicated dual-arm training data—each arm is trained individually, and the factor graph enforces constraints like relative pose and synchronization at test time.
Xu's team also introduced a metric called "temporal ratio" to diagnose the video-action gap. They found that during reaching phases, the model relies heavily on future trajectory predictions (high temporal ratio), but during grasping, attention shifts to the current frame for tactile and visual feedback. This insight enables policy guidance: dynamically adjusting the model's attention between future prediction and current observation to improve success on out-of-distribution tasks.
During the Q&A, Xu addressed the role of failure data. He noted that most video models are trained on successful human demonstrations, lacking failure examples. Once planning algorithms can leverage failure predictions, such data will become valuable for safety boundaries. He also argued for keeping world models and policies separate: world models need to ingest massive, diverse data (including failures) to understand physics, while policies benefit from high-quality success data for fine-tuning. Merging them, he said, would increase pre-training convergence difficulty.
For founders and operators, Xu's work highlights a critical bottleneck in embodied AI: the action side is data-starved compared to vision. Building systems that can compose skills at test time—rather than relying on end-to-end generation—may be a more practical path to robust deployment. The factor graph approach offers a way to reuse existing skill libraries without retraining for every new task combination.
Source: 雷峰网.