TWO EYES. TWO HANDS.

ESTHER.

Egocentric Stereo Hand Estimation
and Reconstruction in the Wild

Hongyu Ma · Hairong Qu · Shiqi Zhao · Yongsong Yang · Peng Yin

City University of Hong Kong

Read paper ↗Watch demo ▶Code Coming soon Dataset Coming soon

Image reprojection ↔ Hand meshes in a 3D sceneDownload MP4 ↓

01 / OVERVIEW

Hand motion,
in its real context.

Human dexterity is guided by two eyes watching two hands. ESTHER brings this binocular perspective to hand reconstruction from a moving, head-mounted stereo rig.

Calibrated stereo geometry, temporal context, and a query-gated decoder work together to estimate MANO pose, hand shape, and absolute wrist translation. A labeling and training loop scales supervision from an initial optimized subset to diverse recordings in the wild.

Designed for wearable capture, with evaluation across devices, missing camera views, and dropped frames.

02 / ESTHER3D

Everyday motion.
Diverse environments.

In-the-wild training video paired with
independent motion capture evaluation.

194 h

In-the-wild source video

Industries

308

Real venues

≈10 h

Motion capture testing

Undistorted factory image with hand keypoints alongside reconstructed hand meshes — Fine articulation during small-part handling. Undistorted image and keypoints (left); hand meshes in scene context (right).

A benchmark with two complementary parts

Training recordings carry model-generated pseudo labels. The test collection uses motion capture ground truth for independent metric evaluation: 6 participants, 50 tasks, and 14 hand action types.

Labels represent hand pose and shape, absolute wrist translation, stereo keypoint projections, and bounding boxes, together with calibrated camera parameters.

Dataset release: coming soon ↗

Original paper figure showing ESTHER3D distribution across six industries — Dataset composition from the paper.

View motion capture examples ＋

Studio motion capture sequences with MANO ground truth projected into stereo images — Motion-capture-derived MANO ground truth projected into the left and right views.

03 / METHOD

Geometry meets motion.

Three components, one sequence of reconstructed hands.

See in stereo

Calibrated rays and cross-view attention connect image evidence across both eyes. Joint queries extract features for the hand.

Read the motion

A masked temporal encoder combines joint features and optical flow to model hand and camera motion across frames.

Fuse by query

Pose, shape, and wrist queries select complementary image and motion features to recover MANO parameters and translation.

THE LABELING LOOP

Better initialization.
Scalable refinement.

Optimize an initial 10 hours of video with the calibrated teacher, filter the labels, and train the first ESTHER model. Model predictions then initialize the same optimization on more samples.

Predict→Refine→Filter→Retrain

About 2,000 optimization steps initially; 200 with model initialization. Deployment and testing use the final model.

Explore the labeling pipeline ＋

Calibrated labeling pipeline from 2D detections and stereo triangulation to MANO fitting — The calibrated optimization pipeline used to construct training supervision.

04 / EVALUATION

Accuracy.
Temporal consistency.

Controlled architecture comparison on
the ESTHER3D motion capture test set.

17.7 mm

Wrist-relative MPJPE

3.3 m/s²

Temporal jitter

Read full results, protocols, and limitations in the paper ↗

05 / RESOURCES

Explore ESTHER.

01 / PAPER ↗

Full paper

Main text and supplementary material in one PDF.

Open PDF

02 / CODE

Implementation

Code release and usage instructions.

Coming soon

03 / DATASET

ESTHER3D

Dataset access and accompanying documentation.