From Human Hand to Robot Hand: What Recent Research Says About Retargeting Data-Glove Motion

In short: a data glove records how a person's fingers move, but a robot hand has different joints, finger lengths and limits, so that motion cannot be replayed directly. The step that converts one into the other is called retargeting, and it decides how good the demonstration data is. Recent papers split into three approaches: optimisation, learned mappings that need less tuning, and methods that add touch. Several of them use MANUS gloves, so the findings apply directly to Physical AI labs that collect data this way.

If you are new to the topic, our earlier guides cover how MANUS gloves train robot hands and which dexterous hands launched in 2026. This article goes one level deeper into the research question underneath them.

Why a glove recording cannot be replayed as it is

A MANUS glove, for example, exposes 25 landmarks per hand. A robot hand might have 12, 16 or 20 joints, thumbs in different places and fingers of different lengths. Copying joint angles one to one makes the robot's fingertips land in the wrong place, so a pinch misses the object.

A September 2026 paper, DexTaG, states the problem plainly: glove-based capture can collect demonstrations at scale, but human and robot hands differ in kinematics, so recorded motions cannot simply be replayed. The authors say this matters most in contact-rich work such as in-hand reorientation and tool use.

Approach 1: optimise fingertip geometry

The ByteDexter paper (July 2025) pairs a 20-degree-of-freedom linkage-driven robot hand with an optimisation-based retargeting method. The glove is a MANUS Quantum Metaglove, sampled at 120 Hz, and the MANUS SDK provides 25 landmarks per glove. A Meta Quest 3 controller on the back of the glove tracks the wrist.

Instead of matching joint angles, the method matches vectors. It compares 15 vectors that run from each fingertip to its own knuckle joint on the human and robot hands, which accounts for the robot palm not being human-sized. The objective weights pinch grasps differently from power grasps and adds a smoothness term. The hand is commanded in a 100 Hz loop and the arm in a 50 Hz loop. The team tested it on in-hand manipulation and on tidying a makeup table with nine objects.

Approach 2: learn the mapping

Optimisation solves a small problem every frame and needs hand-tuned objectives. Two 2026 papers try to remove that tuning.

HumDex (March 2026) uses IMU-based gloves, either Vdhand or MANUS, and a learned retargeter. The input is only the 3D positions of the five fingertips in the glove's wrist frame, 15 numbers in all. The output is 20 robot-hand joint angles. Against a per-frame optimisation baseline, the authors report teleoperation successes out of 30 trials of 20 versus 14 on a scanner-triggering task and 29 versus 23 on doll grasping, with the baseline slightly ahead on hanger stabilisation (21 versus 20). The same system also supports a two-stage imitation learning recipe: pre-train on diverse human motion, then fine-tune on robot data.

AnyDexRT (July 2026) is calibration-free. Operators wore MANUS Quantum Metagloves with a Vive Tracker on the arm. The method learns fingertip correspondences without labelled supervision, uses a small amount of human guidance to focus on the task-relevant regions, and adds a contact classifier to refine pinch poses. The authors tested seven simulated hands (Inspire, Ability, XHand, Wuji, Allegro, Leap and Shadow) and a real Wuji hand on a Flexiv Rizon 4 arm. Reported results: about 90.2% local motion consistency against 59.8% for GeoRT and 52.2% for an optimisation baseline, about 293 Hz, and a 62.0% pinch success rate against 39.6% and 29.2%. These are the authors' own numbers on their own setup.

Approach 3: bring touch into the mapping

Geometry alone can place a fingertip correctly and still lose the grip. Two papers add contact information.

• RoboPaint (February 2026) records human demonstrations in standardised rooms with RGB-D video, 29-degree-of-freedom glove joint angles and 14 channels of tactile signal. Its retargeting objective has a kinematic term plus a tactile term that pulls each glove contact point toward the matching point on the robot hand, with more weight where the person pressed harder. The glove is a custom build, not a MANUS product. The authors report an 84% success rate for retargeted trajectories across 10 object tasks, and 80% average success across three tasks for a Pi0.5 policy trained only on the generated data.
• DexTaG (September 2026) uses the glove's tactile signals to steer reinforcement learning toward the contact pattern the human actually made. It trains one retargeter per object across trajectories, then distils it into a controller that needs no tactile input at run time. It shows contact-rich behaviour on marker-pen and hammer tasks, and the abstract gives no numeric results.

A related hardware line is DOGlove (February 2025), an open-source glove that captures 21 degrees of freedom and returns force and vibration to the operator for under US$600 in parts. Its experiments without visual feedback indicate that haptic feedback is critical to task performance.

What the papers agree on

• The demonstration is only as good as the mapping. Tracking quality and retargeting quality both limit the final policy.
• Wrist tracking matters. ByteDexter and AnyDexRT add a separate controller or tracker for the wrist and arm, because a glove measures fingers.
• Tuning is the hidden cost. The learned and calibration-free methods exist because per-hand objective tuning does not scale across robot hands.
• Touch is the next layer. Papers that add contact signals aim to keep the grip, not only the finger shape.

These are research prototypes, and each paper tests on its own hands and tasks, so results are not directly comparable between them.

What a lab can take from this

These are practical suggestions based on the papers, not claims from them.

1. Log the raw glove stream, not only the retargeted output. A better retargeter can then be applied to old recordings.
2. Keep the robot hand model in version control. Retargeting depends on the hand's geometry.
3. Evaluate on pinch and grasp success, not only joint error. HumDex, AnyDexRT and RoboPaint all report task outcomes such as success rates or pinch success.
4. Add wrist tracking from the start and calibrate it the same way for every operator.
5. Plan for touch. Even if your first gloves do not record contact, keep the data format ready for it.

Where Aedu AI fits

Aedu AI designs, supplies and installs Physical AI Labs and XR Labs in Singapore, Malaysia and across Southeast Asia. We supply MANUS Metagloves, set up NVIDIA Isaac Teleop and Isaac Lab, connect your robot through ROS 2, and train your team to record clean data. For how a university lab combines VR and physical practice, see Train in VR, Learn in the Real Room. To plan a lab, get in touch.

Key points

• Retargeting converts glove motion into robot-hand commands and decides the quality of the demonstration data.
• ByteDexter matches fingertip-to-knuckle vectors using a MANUS Quantum Metaglove sampled at 120 Hz.
• HumDex and AnyDexRT learn the mapping to cut manual tuning; both report gains over optimisation baselines in their own tests.
• RoboPaint and DexTaG add tactile signals so the grip, not only the finger shape, is preserved.
• Labs should log raw glove data, track the wrist and judge retargeting by task success.

FAQ

What is hand retargeting?

It is the step that converts a human hand's recorded motion into commands a robot hand can follow, given the robot's different joints, finger lengths and limits.

Why can't I copy the glove's joint angles to the robot hand?

Human and robot hands differ in structure, so equal angles put the fingertips in different places. Methods therefore match fingertip positions or contact points instead.

Do I need a tactile glove?

Not to start. The papers that use touch show it helps keep the grip in contact-rich tasks, but most hand-teleoperation setups begin with finger tracking only.

Does a MANUS glove work with any robot hand?

It works with any hand that has a retargeter, which in practice means a model of the hand and a mapping method. The papers above cover hands such as Wuji, Inspire, Allegro and Shadow.

Is learned retargeting better than optimisation?

The authors of HumDex and AnyDexRT report gains in their own tests, with one task where the baseline was slightly ahead. The right choice depends on the hand and the task.

Can Aedu AI help set up glove-based data collection?

Yes. We design Physical AI Labs with MANUS gloves, Isaac Teleop and Isaac Lab, and train teams to record episodes. See our Physical AI Lab page.

Sources

• DexTaG: Tactile-as-Guidance in Reinforcement Learning for Dexterous Manipulation (arXiv, 27 September 2026): https://arxiv.org/abs/2609.33882
• AnyDexRT: Calibration-Free Dexterous Hand Retargeting with Few-Shot Human Guidance (arXiv, 9 July 2026): https://arxiv.org/abs/2607.08341
• HumDex: Humanoid Dexterous Manipulation Made Easy (arXiv, March 2026): https://arxiv.org/abs/2603.12260
• RoboPaint: From Human Demonstration to Any Robot and Any View (arXiv, February 2026): https://arxiv.org/abs/2602.05325
• Dexterous Teleoperation of 20-DoF ByteDexter Hand via Human Motion Retargeting (arXiv, July 2025): https://arxiv.org/abs/2507.03227
• DOGlove: Dexterous Manipulation with a Low-Cost Open-Source Haptic Force Feedback Glove (arXiv, February 2025): https://arxiv.org/abs/2502.07730

All papers are arXiv preprints and have not necessarily been peer reviewed. Figures are the authors' own reported results. The cover image is an Aedu AI illustration.