Empowering Embodied Manipulation: A Bimanual-Mobile Robot Manipulation Dataset for Household Tasks

News

• 2024/08: We have collected data for 5 new manipulation tasks.

Play with a Rubik's Cube

Get a cup of hot water

Charge the iPhone

Hand a cup of tea to a person.

Pick up a cup and place a tea bag inside

Abstract

The advancements in embodied AI are increasingly enabling robots to tackle complex real-world tasks, such as household manipulation. However, the deployment of robots in these environments remains constrained by the lack of comprehensive bimanual-mobile robot manipulation data that can be learned. Existing datasets predominantly focus on single-arm manipulation tasks, while the few dual-arm datasets available often lack mobility features, task diversity, comprehensive sensor data, and robust evaluation metrics; they fail to capture the intricate and dynamic nature of household manipulation tasks that bimanual-mobile robots are expected to perform. To overcome these limitations, we propose BRMData, a Bimanual-mobile Robot Manipulation Dataset specifically designed for household applications. BRMData encompasses 10 diverse household tasks, including single-arm and dual-arm tasks, as well as both tabletop and mobile manipulations, utilizing multi-view and depth-sensing data information. Moreover, BRMData features tasks of increasing difficulty, ranging from single-object to multi-object grasping, non-interactive to human-robot interactive scenarios, and rigid-object to flexible-object manipulation, closely simulating real-world household applications. Additionally, we introduce a novel Manipulation Efficiency Score (MES) metric to evaluate both the precision and efficiency of robot manipulation methods in household tasks. We thoroughly evaluate and analyze the performance of advanced robot manipulation learning methods using our BRMData, aiming to drive the development of bimanual-mobile robot manipulation technologies.
Descriptive Alt Text

Autonomous Skills

Bottle Pick

Bottle Handoff

Plate Place

Cup Place

Fruit Handover

Wine Wipe

Single Fruit Pick

Multiple Fruits Pick

High Five

Garbage Recycle

Failures

Using the BRMData dataset, we have observed several instances of failure. These documented failures provide critical insights into the challenges and limitations inherent in integrating robotic hardware and Embodied AI technologies.

BibTeX

@inproceedings{zhang2024empowering,
  author    = {Zhang, Tianle and Li, Dongjiang and Li, Yihang and Zeng, Zecui and Zhao, Lin and Sun, Lei and Chen, Yue and Wei, Xuelong and Zhan, Yibing and Li, Lusong and He, Xiaodong},
  title     = {Empowering Embodied Manipulation: A Bimanual-Mobile Robot Manipulation Dataset for Household Tasks},
  booktitle = {arXiv},
  year      = {2024},
}