At the 2026 Robotics: Science and Systems (RSS) conference, a workshop roundtable turned into a heated debate on three fundamental questions for embodied AI: should world models and policy models be unified or separate? Should control rely on pixel-level video generation or action-level grounding? And should training data come from internet/human video or embodied robot data?
The panel included Danfei Xu (Georgia Tech & Nvidia), Sherry Yang (Google & NYU), Laura Smith (Physical Intelligence), Max Li (Nvidia), and Li Chen (OpenDriveLab). The moderator posed three topics, splitting the panel into opposing camps each round.
Round 1: Unify or Separate World and Policy Models?
Laura Smith and Max Li argued for unification, citing the trend toward everything-in-one models. Smith noted that MoE architectures already pack hundreds of experts into one network, so why not combine world and policy? Li added that the ultimate goal is a single model that both understands the world and acts, but he acknowledged a gap between what video models can imagine and what robots can execute, requiring better representation learning.
Opposing them, Li Chen, Sherry Yang, and Danfei Xu argued for separation. Chen stressed real-time efficiency: policy models must run fast, and merging a heavy world model would slow them down. Yang pointed out that most models are underfit; world models need failure data to learn physics, while policy models need success data—combining them causes conflicting objectives. Xu highlighted debugging clarity: when a robot fails, you can pinpoint whether the world model or policy is at fault. Audience voting favored the separate camp.
Round 2: Control via Pixels or Actions?
This round reshuffled alliances. Max Li, Laura Smith, and Li Chen backed pixel-level video generation as the most interpretable interface for humans. Li compared high-fidelity video to a window into the model's mind, essential for blame assignment when things go wrong.
Danfei Xu and Sherry Yang argued for action-level grounding. Xu noted that video is only a partial slice of reality—humans rely on touch and force feedback, not mental imagery, when grasping in the dark. Yang admitted she had previously favored pixels but recent experiments showed that even massive models fail on contact-rich tasks; she now advocates injecting semantic and physical priors into video generation. The round ended without a clear winner, but consensus emerged that video and action must be combined, not pitted against each other.
Round 3: Internet/Human Video vs. Embodied Robot Data?
The final debate pitted Danfei Xu, Sherry Yang, and Max Li (internet data) against Li Chen and Laura Smith (embodied data). The internet camp argued that human video provides massive scale and diverse interactions, boosting generalization. Xu claimed that pure robot data is too limited to cover long-tail scenarios. Yang added that humans are the most powerful physical agents, and human video can be treated as robot data via end-to-end control or pose alignment.
The embodied camp countered that robot data contains real physics, action feedback, and contact forces—essential for learning contact dynamics. Chen expressed skepticism about internet video's direct utility for precise manipulation; it helps with sub-goal planning but not low-level control. Audience voting gave the internet camp 61% vs. 39%.
Takeaways for Founders and Operators
- Modular design wins for now. Separating world and policy models enables faster iteration, easier debugging, and better real-time performance. Startups should resist the temptation to build monolithic models until the field matures.
- Don't over-index on video quality. For contact-rich tasks, action-level grounding and physical priors matter more than pixel fidelity. Invest in tactile sensing and force feedback.
- Hybrid data strategies are non-negotiable. Internet video provides scale and diversity; embodied data provides precision. Build pipelines that leverage both, with careful attention to data alignment and failure case coverage.
- The debate is far from settled. The fact that top researchers disagree on fundamentals signals that the field is still pre-paradigm. Founders should stay agile, hedge bets, and avoid premature architectural commitments.
Source: 雷峰网.