Dobb-E

What is Dobb-E?

Dobb·E is an open-source framework aimed at teaching robots household tasks through imitation learning. It addresses the limitations in home robotics by providing an affordable solution for collecting demonstrations using a tool called the Stick. This tool is created with a $25 Reacher-grabber stick, 3D printed parts, and an iPhone. Dobb·E utilizes the Stick to gather data from the Homes of New York (HoNY) dataset, which includes 13 hours of interactions in 22 different homes in New York City. The framework trains a representation learning model called Home Pretrained Representations (HPR) based on the ResNet-34 architecture and self-supervised learning objectives, enabling robots to perform new tasks in various environments.

Who created Dobb-E?

Dobb-E was created by Nur Muhammad Mahi Shafiullah, Anant Rai, Haritheja Etukuru, Yiqian Liu, Ishan Misra, Soumith Chintala, and Lerrel Pinto. The framework was launched on November 28, 2023, as an open-source project aiming to revolutionize household robotic manipulation through imitation learning. It leverages tools like "The Stick" for data collection and features the "Home Pretrained Representations" model for initializing robot policies in new environments.

What is Dobb-E used for?

• Teaching robots household tasks through imitation learning
• Addressing limitations of current home robotics by providing a cheap and ergonomic solution for collecting demonstrations
• Utilizing the Stick tool to collect data from the Homes of New York dataset
• Training a representation learning model called Home Pretrained Representations (HPR)
• Achieving an 81% average success rate in solving novel tasks within 15 minutes
• Providing access to pre-trained models, code, and documentation through GitHub
• Initiating a robot policy for performing new tasks in novel environments
• Deploying Home Pretrained Representations (HPR) to initialize policies for new tasks
• Demonstrating the ability to learn new tasks in 20 minutes
• Improving progress in home robotics by enabling easy collection of robot demonstrations
• Collecting demonstrations for robots using a tool called the Stick
• Accessing pre-trained models, code, and documentation through GitHub
• Providing insights into the methodology and results in the paper 'On Bringing Robots Home'
• Using the Stick tool to collect data from the Homes of New York (HoNY) dataset
• Utilizing the Stick tool constructed with a Reacher-grabber stick, 3D printed parts, and an iPhone for data collection
• Utilizing the HPR model based on ResNet-34 architecture and self-supervised learning objectives
• Demonstrating the ability to learn a new task in 20 minutes

How to use Dobb-E?

To use Dobb·E for teaching robots household tasks through imitation learning, follow these steps:

1. Obtain the Model: You can get the Dobb·E model at Huggingface or by using PyTorch Image Models (TIMM) with a few lines of code.

2. Collect Demonstrations with The Stick: Use the Stick, a tool created from a $25 Reacher-grabber stick, 3D printed parts, and an iPhone, to collect data from the Homes of New York (HoNY) dataset. This dataset includes RGB and depth videos along with action annotations for the gripper's pose and opening angle.

3. Train the Home Pretrained Representations (HPR) Model: Dobb·E leverages the collected data to train the HPR model, which is a ResNet-34 architecture trained on the HoNY dataset using self-supervised learning objectives.

4. Initialize Robot Policy: During deployment, the HPR model is used to initialize a robot policy for performing new tasks in different environments. The trunk of the policy consists of the pretrained ResNet-34 model followed by two linear layers.

5. Access Resources: Dobb·E provides access to pre-trained models, code, and documentation through GitHub. Additionally, a paper titled "On Bringing Robots Home" offers further insights into the methodology and results of Dobb·E.