Scientists have developed a system for training humanoid robots

A research team from Carnegie Mellon University (CMU) has created the Human2Humanoid (H2O) system. It allows people to control a full-size humanoid robot in real time using only an RGB camera.

To develop H2O, experts used the Reinforcement learning (RL) method, which reproduces the human learning strategy based on trial and error. Using a system of rewards and penalties, a humanoid robot can learn from its mistakes and successes, independently determining the most effective strategies to achieve its goals.

“H2O is a reinforcement learning (RL)-based platform that facilitates real-time full-body telesurgery of humanoid robots using only an RGB camera.”

Tyran He, CMU major

The learning process begins with adapting human movements to the physical capabilities of humanoid robots using the sim-to-data method. It ensures that the robot's physical limitations are respected when performing movements. The improved motion dataset is then used to train an RL-based motion simulator in the simulation. The resulting result is then transferred to the robot without the need for further adjustment.

To simulate the operator's movements, images obtained using an RGB camera and a 3D pose estimator were used.

Videos show the H2O-powered robot kicking a ball, throwing a box, dodging while in a boxing stance, and walking with a stroller. This is the first public presentation of a remote-controlled humanoid robot operating in real time.

In the future, CMU researchers plan to explore the possibility of using other training methods, including force feedback and conversational feedback, to expand H2O's functionality. In addition, a promising area for further research is the inclusion of lower body tracking, which would allow robots to replicate more complex human movements, such as dancing or performing sports exercises.