WinDEX-Net
9.50 FPSAnnotation refinement
and learned reconstruction
For dexterous teleoperation
and policy learning.
Low-cost stereo capture, geometric annotation, and fast reconstruction of metric hand trajectories.
Training annotations
Human-annotated 2D test set
Offline reconstruction
Causal streaming
Everyday manipulation across 13 activity and capture categories adapted from O*NET, from food preparation and repair to farming, healthcare, and education.
1,000+ scenes
200+ operators
5,000+ tasks
Original recordings · no annotation overlaysTraining labels are kept separate from evaluation references: 20 hours of held-out, human-annotated in-the-wild 2D data and independent optical motion-capture ground truth.
Stereo observations constrain depth. Hand priors constrain articulation. Temporal refinement and camera trajectories recover consistent motion beyond the moving camera.
Real scene geometry. Continuous hand motion.
WinDEX-Net learns directly from fisheye stereo. Model-assisted annotation and retraining expand supervision; distillation brings reconstruction to offline and streaming applications.
Annotation refinement
and learned reconstruction
Bidirectional context
for accurate reconstruction
Continuous motion estimates
without future frames
Model throughput. WinDEX-Net and WinDEX-Window use two crop iterations. Method and evaluation in the paper
Two interfaces connect reconstructed motion to dexterous manipulation: Hand2Teleop for live control and Hand2Policy for trajectory-guided reinforcement learning.
Live arm–hand teleoperation using head-worn stereo cameras and IMU for operator tracking.
Grasp, lift, hold, and place. Reconstructed human trajectories guide reinforcement learning.
In-the-wild image consistency is evaluated against human annotations; 3D accuracy is evaluated against independent optical motion capture.
WinDEX-Window on held-out, human-annotated 2D references.
WORLD-FRAME JOINT ERROR37.29 mmWinDEX-Window with visual–inertial odometry, evaluated against optical ground truth.
Position errors in mm; RTE in %.
| Method | W-MPJPE ↓ | WA-MPJPE ↓ | RTE ↓ |
|---|---|---|---|
| HaWoR (pretrained) | 100.92 | 44.19 | 3.10 |
| HaWoR (fine-tuned) | 97.65 | 40.91 | 2.83 |
| WinDEX-Window + cuVSLAM | 37.29 | 17.69 | 2.00 |
Mean simulated grasp–lift–place success, with the same training settings and budget.

Metric 3D Hand Motion Reconstruction in the Wild for Dexterous Teleoperation and Policy Learning
WinDEX integrates geometric–temporal optimization, end-to-end fisheye-native reconstruction, and efficient distilled models to recover metric 3D hand motion from in-the-wild egocentric stereo videos. Visual–inertial SLAM places the motion in a world frame. Model-assisted annotation and retraining form a data–model flywheel, producing scalable supervision while keeping human-annotated 2D and optical motion-capture evaluation references independent. The resulting representations support reinforcement learning and real-time arm–hand teleoperation.