WinDEX · Research project

Metric 3D
hand motion.
In the wild.

For dexterous teleoperation
and policy learning.

Low-cost stereo capture, geometric annotation, and fast reconstruction of metric hand trajectories.

THE RESEARCH FILM01:30 · SOUND & SUBTITLES
Capture → Reconstruction → Robot action

Hongyu Ma1, Hairong Qu1, Shiqi Zhao1, Yongsong Yang1, Kai Shi1, Jiahao Ai1, Xin You2, Peng Yin1

1 City University of Hong Kong2 CyberOrigin

1,000h

Training annotations

20h

Human-annotated 2D test set

43.61FPS

Offline reconstruction

171FPS

Causal streaming

01 In-the-wild data02 World-frame reconstruction03 Efficient inference04 Robot manipulation
01 / WINDEX-HAND

Real hands.
A world of activities.

Everyday manipulation across 13 activity and capture categories adapted from O*NET, from food preparation and repair to farming, healthcare, and education.

1,000+ scenes

200+ operators

5,000+ tasks

Original recordings · no annotation overlays
INDEPENDENT EVALUATION

Training labels are kept separate from evaluation references: 20 hours of held-out, human-annotated in-the-wild 2D data and independent optical motion-capture ground truth.

02 / GEOMETRIC–TEMPORAL ANNOTATION

From pixels
to world-frame motion.

Stereo observations constrain depth. Hand priors constrain articulation. Temporal refinement and camera trajectories recover consistent motion beyond the moving camera.

01 / OBSERVE

EXPLORE IN 3D

Moving hands. A fixed world frame.

Real scene geometry. Continuous hand motion.

03 / LEARNED RECONSTRUCTION

Accurate offline.
Ready in real time.

WinDEX-Net learns directly from fisheye stereo. Model-assisted annotation and retraining expand supervision; distillation brings reconstruction to offline and streaming applications.

RECONSTRUCTION TEACHER

WinDEX-Net

9.50 FPS

Annotation refinement
and learned reconstruction

OFFLINE MODEL

WinDEX-Window

43.61 FPS

Bidirectional context
for accurate reconstruction

CAUSAL MODEL

WinDEX-Stream

171 FPS

Continuous motion estimates
without future frames

Model throughput. WinDEX-Net and WinDEX-Window use two crop iterations. Method and evaluation in the paper

04 / ROBOT APPLICATIONS

Human motion.
Robot action.

Two interfaces connect reconstructed motion to dexterous manipulation: Hand2Teleop for live control and Hand2Policy for trajectory-guided reinforcement learning.

EVALUATION

Measured on
held-out recordings.

In-the-wild image consistency is evaluated against human annotations; 3D accuracy is evaluated against independent optical motion capture.

IN-THE-WILD REPROJECTION14.5 pixels

WinDEX-Window on held-out, human-annotated 2D references.

WORLD-FRAME JOINT ERROR37.29 mm

WinDEX-Window with visual–inertial odometry, evaluated against optical ground truth.

World-frame motion reconstruction

Position errors in mm; RTE in %.

MethodW-MPJPE ↓WA-MPJPE ↓RTE ↓
HaWoR (pretrained)100.9244.193.10
HaWoR (fine-tuned)97.6540.912.83
WinDEX-Window + cuVSLAM37.2917.692.00

Trajectory-guided policy learning

Mean simulated grasp–lift–place success, with the same training settings and budget.

WiLoR references8%
HaWoR references17%
WinDEX references61%
First page of the WinDEX paper
PAPER & RESOURCES

WinDEX

Metric 3D Hand Motion Reconstruction in the Wild for Dexterous Teleoperation and Policy Learning

Read the abstract +

WinDEX integrates geometric–temporal optimization, end-to-end fisheye-native reconstruction, and efficient distilled models to recover metric 3D hand motion from in-the-wild egocentric stereo videos. Visual–inertial SLAM places the motion in a world frame. Model-assisted annotation and retraining form a data–model flywheel, producing scalable supervision while keeping human-annotated 2D and optical motion-capture evaluation references independent. The resulting representations support reinforcement learning and real-time arm–hand teleoperation.

WINDEX / RECORDINGS