Scaling humanoid object carrying learning by expanding a few real videos with AI
- What
- Proposed a method (PRISM) where AI generates various virtual videos based on a few real videos of a person carrying objects, and uses them for robot learning.
- Key result
- Reports that a single control policy trained this way successfully enabled a real humanoid robot to pick up, transport, and place boxes, drums, kegs, and balls without additional real-world learning.
- Why it matters
- Significant in that it demonstrates the potential to increase training data using a few real videos in situations where collecting large amounts of real robot demonstration data is difficult.
Background
Teaching skills like object carrying to humanoid robots by showing them human motions is considered a promising path toward general-purpose robots. However, gathering a variety of high-quality videos where the human's entire body is clearly visible and interactions with objects are not occluded poses a practical barrier.
Method
First, using a small number of real videos as input, video-to-video (V2V) generation technology is applied to create hundreds of counterfactual human-object interaction videos. Next, human and object motions are reconstructed in a simulation space based on contact points with the object, and converted into physically plausible trajectories tailored for the robot. By leveraging within-category diversity, a single policy that generalizes to previously unseen objects is trained.
Main results
The trained policy was applied to a real robot, and it is explained that the operations were performed without additional fine-tuning in the real environment. It is reported that the robot performed picking, transporting, and placing actions for new objects, different sizes, and different initial placements using only information from the onboard depth sensor.
Limitations & open points
The abstract does not provide specific figures such as success rates, quantitative comparisons with other methods, or costs and time required for training. Furthermore, it is difficult to judge solely from the abstract how faithfully the quality of the generated videos reflects real situations or whether the same performance can be achieved in more complex tasks or different environments.
Glossary
- Humanoid — A robot that has a head, torso, arms, and legs similar to a human.
- Loco-manipulation — The ability to simultaneously perform walking or moving actions and grasping and manipulating objects.
- Counterfactual video — Newly created videos assuming 'what if this situation had occurred' even though it did not actually happen.
- Video-to-video (V2V) generation — An AI technology that takes an existing video as input and generates a new video with changed shapes or situations.
- Retargeting — The task of translating human motions to fit the body structure and joints of a robot.
- Real-to-sim-to-real — A method of transferring real-world data to a virtual simulation for learning and then applying it back to a real robot.
- Policy — A learned rule or model that determines what action a robot should take based on its surroundings.
- Onboard depth sensor — A sensor mounted on the robot body that measures the distance to objects.
This summary was written by AI from the paper's abstract and reviewed by the editor. Summaries on this page were translated by AI from the editor-reviewed Korean version. Check the original for exact figures and context.
This page compiles published papers, official announcements and press reports for information only. Paper summaries are written by AI from the abstract and reviewed by the editor; they may differ from the original, so check the paper for anything important. Nothing here is a recommendation to buy or sell any security or investment advice. Career moves cite official announcements or major press reports; some career records come from Wikidata (CC0). Summaries on this page were translated by AI from the editor-reviewed Korean version.
