Aedu AI
All posts

Physical AI9 min read

Train in VR, Learn in the Real Room: How a University Humanoid Lab Combines Virtual Data and Physical Practice

At a Singapore university's robotics lab, operators wear a VR headset and MANUS data gloves to create training data in virtual scenes, while a Unitree humanoid learns in a real home-style room. How the pattern works, how the industry does it, and where XR labs fit in.

Aedu AI Team
Train in VR, Learn in the Real Room: How a University Humanoid Lab Combines Virtual Data and Physical Practice

In short: a humanoid robot learns household skills best when it gets two kinds of practice. Virtual scenes give an operator a fast, safe, repeatable place to produce training data with a VR headset and data gloves. A real room gives the robot the friction, lighting and contact that no simulation fully captures. A university robotics lab in Singapore combines both: operators wear a VR headset and MANUS data gloves, and a Unitree humanoid learns in a physical, home-style room. Aedu AI supplied the MANUS Metagloves for the set-up.

This article explains how that pattern works, how the wider industry trains robots with virtual and physical data, and what it means for universities planning a Physical AI lab or an XR lab.

What the university lab does

A university robotics lab in Singapore is working on humanoid data collection: teaching a robot household tasks such as making a bed or tidying a room. The approach is built around two worlds:

  • A virtual world for creating data. Operators wear a VR headset and MANUS data gloves. Their hand and finger movements are captured precisely and become training data.
  • A physical world for learning. The lab is furnished like a real home, with a bed and everyday objects, so the Unitree humanoid with dexterous hands learns in the kind of room it will work in.

Aedu AI supplied MANUS Metagloves for the lab's teleoperation set-up. More delivered labs are on our projects page.

Why use two worlds

Our reading of this design, based on how Physical AI labs are planned, is that each world covers a weakness of the other:

  • Virtual scenes are cheap to repeat. A reset takes a click, not a person tidying a room. Operators can try risky or unusual situations without damaging the robot or the room, and many scenes and objects can be swapped in quickly.
  • Real rooms are honest. Cloth folds, bedding sags, doors stick and lighting changes. Tasks such as making a bed depend on contact and softness, which are hard to simulate faithfully. Practice in a physical room keeps the robot's learning tied to reality.
  • Hand data is what links them. Whatever the world, the quality of the recorded finger movement decides how much a robot hand can learn from it.

How the industry trains with virtual and physical data

The university lab's pattern is part of a wider shift. These are the main approaches in published research and in NVIDIA's tools. The figures are the authors' own reported results.

1. Collect in simulation, then fine-tune on the real robot. In Pre-training Visual Dexterity in Simulation (Stanford, MIT and Scale AI, August 2026), five operators collected 75 hours of demonstrations in simulation across six scenes and hundreds of objects, which pre-trained a dexterous-hand policy. A separate real-world teleoperation system then supplied data for post-training. The authors report better results than training from scratch.

2. Mix simulated and real data in one training run. Sim-and-Real Co-Training (16 authors, 2025) blends simulated and real demonstrations in a single training run. Across a robot arm and a humanoid, the authors report that adding simulation data lifted real-world task performance by 38% on average.

3. Multiply a few human demonstrations in simulation. MimicGen takes a small set of human demonstrations and adapts them to new scenes. From about 200 human demonstrations it generated more than 50,000 new ones across 18 tasks. NVIDIA's Isaac Lab Mimic brings this approach into Isaac Lab.

4. Bring the real scene into the virtual one. RialTo from MIT CSAIL (2024) lets a user scan a room with a phone, builds a digital twin in a simulator, and uses a few real demonstrations to start training. The researchers say policies trained this way performed 67 percent better than imitation learning given the same demonstrations. Google's Gemini 4 Argon (October 2026) takes a similar real-to-sim step and reports that simulated episodes raised one real-world task from 56% to 75% success. We covered it in Teleoperation or Human Video?.

5. Use VR as the operator's window. Open-TeleVision (UC San Diego and MIT) lets an operator look around a robot's workspace in stereo through a VR headset while the robot copies arm and hand motion, tested on a Unitree H1. Holo-Dex (NYU) and Bunny-VisionPro (HKU and UC San Diego) use VR headsets for dexterous hand teleoperation, the latter with low-cost haptic feedback. VR-DAgger (May 2026) replays a robot's uncertain moments as clips inside VR so the operator can correct them; the authors report gains of up to 23 percentage points over behaviour cloning and about 40% less collection time per sample.

6. Tooling that connects the pieces. NVIDIA's Isaac Lab 2.3 added teleoperation with Meta Quest VR and MANUS gloves, and dexterous retargeting for the Unitree G1 with Unitree and Inspire hands. NVIDIA's Isaac ROS reference workflow describes collecting teleoperated Unitree G1 demonstrations through XR teleoperation, recording them, and fine-tuning a GR00T policy.

For contrast: training without demonstrations. VIRAL trains a Unitree G1 for loco-manipulation entirely in simulation with reinforcement learning and deploys it zero-shot, without real-world fine-tuning, which its authors say approaches expert teleoperation performance. Tasks that need fine finger control, such as dexterous hands handling soft or small objects, generally still rely on human demonstrations, which is where data gloves matter.

Why the quality of hand data decides the result

A human hand and a robot hand are built differently: finger lengths, joint limits and degrees of freedom do not match. Retargeting maps the human motion onto the robot's joints, and any inaccuracy in the captured motion is carried into the training data. Noisy or drifting finger tracking teaches the robot the wrong thing.

MANUS Metagloves use electromagnetic fingertip sensors, with a 25-DoF hand model and 120 Hz sampling, so finger motion stays tracked when a hand closes around an object. Metagloves Pro Haptic adds per-finger vibration feedback. For how teams record demonstrations with them, see How MANUS Gloves Are Used to Train Robot Hands, and for the robot hand side, our guide to dexterous robot hands.

Where XR and Physical AI meet

VR teleoperation and an XR lab share most of their building blocks: a headset, tracking, a GPU workstation and 3D content. That makes the two lines of work natural neighbours:

  • One skill set, two uses. The staff who build VR training scenes for students can build the virtual scenes in which operators record robot data.
  • Virtual scenes you control. AeduVerse, our browser-based XR training platform, lets teachers build 3D lessons without code. The same 3D-scene know-how applies to preparing environments for simulation.
  • Review in a shared space. After collection, teams need to look at scenes and recorded episodes together. A CAVE or projection space lets a class or a team stand inside the scene, and a holographic display lets a small group review a 3D model with real depth, without headsets.
  • A path that grows. Many labs start with simulation only, which needs no robot, then add a robot station. We set out the phases in How to Build a Physical AI Teaching Lab.

What a lab needs to build this pattern

  • Capture: a VR headset and data gloves, with a tracker for the wrist.
  • Virtual side: a GPU workstation running NVIDIA Isaac Sim and Isaac Lab, with Isaac Teleop connecting the headset and gloves.
  • Physical side: a robot, a dexterous hand if the task needs it, cameras, ROS 2, and a room set up for the tasks you want to teach, such as a home-style space.
  • Data pipeline: an episode recorder that keeps video, hand motion and robot state in sync, with storage a training team can use.
  • Review: a way for the team to look at scenes and episodes together.

Where Aedu AI fits

Aedu AI designs, supplies and installs Physical AI Labs and XR Labs in Singapore, Malaysia and across Southeast Asia. Our Physical AI Lab configurations range from a Sim Teleop Lab, which needs no robot, through a Dexterous Teleop Lab with one collaborative arm and a dexterous hand, to a Bimanual Data Lab with two arms or a humanoid upper body. The recorded data belongs to the customer and the system runs on the customer's premises. If you are planning a lab that combines virtual and physical practice, get in touch.

Key points

  • A Singapore university's robotics lab combines virtual and physical practice: operators wear a VR headset and MANUS data gloves to create training data, and a Unitree humanoid learns in a real home-style room. Aedu AI supplied the MANUS Metagloves.
  • The industry pattern is simulation or VR for fast, repeatable data, and real-world data for contact and reality. Published work includes pre-training in simulation then fine-tuning on a real robot, co-training on mixed data, multiplying demonstrations, and building digital twins of real scenes.
  • VR is used as the operator's window in Open-TeleVision, Holo-Dex, Bunny-VisionPro and VR-DAgger. NVIDIA's Isaac Lab 2.3 supports Meta Quest and MANUS gloves for teleoperation.
  • Hand-data quality sets the ceiling: retargeting from a human hand to a robot hand can lose information, so precise, drift-free finger tracking matters.
  • XR labs and Physical AI labs share headsets, tracking and 3D content, so they can be planned together.

FAQ

What is VR teleoperation for robot training?

An operator wears a VR headset and tracking devices, such as data gloves, to control a robot or a simulated robot from the inside of a scene. The operator's movements are recorded as demonstrations that the robot can learn from.

What does a university robotics lab do with MANUS gloves and VR?

Operators wear a VR headset and MANUS data gloves to create training data in virtual scenes, and a Unitree humanoid learns household tasks in a physical room furnished like a home. Aedu AI supplied the MANUS Metagloves for the set-up.

What is sim-to-real in robot learning?

Training in simulation and then transferring what was learned to a real robot. Many teams fine-tune on a small amount of real-world data afterwards, because simulation cannot reproduce every detail of contact, friction and lighting.

Why use data gloves together with a VR headset?

The headset gives the operator a view of the scene, and the gloves capture the finger movement that a robot hand needs to copy. Together they let a person demonstrate delicate tasks naturally.

Do we need a humanoid robot to start?

No. A simulation-only lab, where operators wear gloves and a headset and drive a simulated robot hand in NVIDIA Isaac Lab through Isaac Teleop, teaches most of the workflow. A real arm, a dexterous hand or a humanoid can be added in later phases with the same gloves and software.

How do XR labs connect with Physical AI labs?

Both use headsets, tracking, GPU workstations and 3D content. Staff who build VR training scenes can build virtual scenes for robot data collection, and spaces such as a CAVE or a holographic display help teams review scenes and recorded episodes together.

Who supplies Physical AI labs, VR labs and data gloves in Singapore?

Aedu AI, based in Singapore, designs, supplies and installs Physical AI Labs and XR Labs, and supplies MANUS Metagloves, for universities, polytechnics, research institutes and training centres in Singapore, Malaysia and Southeast Asia.

Sources

  • Physical AI
  • Case Study
  • VR Teleoperation
  • Humanoid Robot
  • Sim-to-Real
  • XR Labs
Keep readingAll posts →

Work with us

Tell us what you want to train, simulate, or visualize, and we will follow up by email.

or email support@aedu.ai