Learning humanoid interactive skills

VICARVideo Imitation with Contact-Aware Retargeting for Humanoid Interactive Skills

Learn from human videos.
Preserve the contacts that make a skill work.

The idea

Human motion. Robot embodiment.
Mimic contacts for the same interaction.

A person kneels at a table-tennis table, one hand on the table edge and one reaching underneath; red dots mark the contacts. Face blurred. — Human video

The gap

Contacts get lost in translation

Contact-rich skills are defined by where, when and how fast contact happens. Those contacts rarely survive the move from a human body to a robot’s.

The retargeted humanoid kneels under the reconstructed table, keeping the same two contacts. — Retargeted

The idea

Contacts come first

VICAR retargets the whole motion while treating contact poses, speed and collision clearance as top priorities, so the interaction stays intact.

The real humanoid robot kneels at the table, touching the same contact points. — Real robot

The result

From video to a real robot

Learned skills transfer zero-shot to a real humanoid and adapt to new contact points at runtime.

Shown: the under-table pickup, from human video to retargeted motion to the real robot.

Read the full abstract

Third-person human videos offer a scalable alternative to task-specific reward engineering, teleoperation, and manually scripted demonstrations for humanoid loco-manipulation. VICAR learns interactive humanoid skills while preserving world-frame contact poses, motion speed, and collision clearance.

From observation to execution

A pipeline built around contact

Reconstruct the scene, preserve the interaction, and learn a reference that adapts.

VICAR pipeline: human videos and scene reconstruction feed contact-aware retargeting, contact-point augmentation, a motion generator, reinforcement learning, and zero-shot deployment on a humanoid robot. — **VICAR overview.** Third-person human video becomes a contact-aware robot reference. Augmented contact points train a motion generator, whose outputs are tracked by a policy on the robot.

Recover

Human motion and task-relevant scene geometry from video.

Retarget

Preserve global contacts, motion speed, and collision clearance.

Augment

Vary contact locations while retaining the demonstrated skill.

Execute

Track contact-conditioned references on the humanoid.

01 — Table tennis

Six serves. Distinct styles.

Watch the humanoid execute each serve, from simple strokes to spin.

6 serves

Choose a serve with the arrows or dots, then press play. Swipe to browse on mobile.

02 — Beyond serving

Reach, support, lift, climb.

Human demonstrations and robot execution, side by side.

4 skills

03 — Explore in 3D

Step inside the motion.

Choose a skill, explore its contact settings, and replay the adapted motion.

Step inside the motion

Explore the saved G1 motion in an interactive 3D scene.

Saved motion

Orbit, zoom, and use the timeline to inspect each motion.

Citation

Acknowledgements

We thank Dhwani Mundra for designing this website and for her careful review of the videos.