arxiv:2607.24744

Data Pyramid for Embodied Manipulation

Published on Jul 27

· Submitted by

Lingdong Kong on Jul 28

Peking University

Upvote

Authors:

Abstract

Multimodal foundation models learned to see and to speak by consuming the whole internet. Embodied agents admit no such shortcut, since they require data that couple observations with physical states and actions. These signals can be provided, to varying degrees, by multiple data sources. In this work, we organize the embodied data ecosystem as a "pyramid" spanning five complementary sources: real-robot data, UMI-style data, egocentric and exocentric data, simulation data, and general vision-language data. We organize the pyramid around the tension between scalability and robot alignment, and further characterize each source in terms of data quality, diversity, reusability, and physical fidelity. We then analyze recent embodied foundation models through the lens of their data recipes, examining how different sources are selected, aligned, and mixed during pretraining. For embodied brain models, vision-language-action models, and world-action models alike, we relate data composition to capabilities in perception, reasoning, planning, action generation, and world prediction. We close by discussing six open challenges: building large-scale tactile datasets, collecting failure and recovery data, developing scalable data-collection pipelines, aligning actions across embodiments, leveraging egocentric data for dexterous manipulation, and designing principled data recipes for robot learning. We hope this work paves the foundation for the design of next-generation embodied systems.

View arXiv page View PDF Project page GitHub 9 Add to collection

Community

ldkong

Paper submitter about 6 hours ago

Multimodal foundation models learned to see and reason from Internet-scale image-text data. Embodied AI, however, requires a fundamentally different ingredient: interaction data that couples perception, physical states, and actions.

But today’s embodied data ecosystem is highly fragmented. Real-robot trajectories, UMI demonstrations, egocentric videos, simulation, and general vision-language data each provide different forms of supervision, yet their roles and trade-offs remain poorly understood.

In this work, we introduce the Embodied Data Pyramid, a unified taxonomy that organizes embodied data into five complementary layers:

Real-Robot Data — highest robot alignment and physical fidelity 🤖
UMI Data — scalable robot-free demonstrations with action supervision 🦾
Egocentric & Exocentric Data — rich human interaction priors from the real world 👓
Simulation Data — scalable robot-oriented interaction with privileged supervision 🕹️
General Data — web-scale perception, reasoning, and semantic knowledge 🌐

Beyond organizing the data landscape, we analyze how recent Embodied Brain Models, Vision-Language-Action Models (VLAs), and World-Action Models (WAMs) combine these heterogeneous data sources, revealing the emerging trends in embodied pretraining recipes.

Finally, we discuss several open challenges, including tactile data, failure and recovery trajectories, scalable data collection, cross-embodiment action alignment, egocentric priors for dexterous manipulation, and principled data recipes for robot learning.

We hope this survey provides a useful roadmap for understanding, collecting, and utilizing data for the next generation of embodied foundation models.

Feedback and discussions are very welcome!