Clarification on Egocentric Image Dataset and Training Script

[juanitapuentes]

Hi :)

I appreciate the work done in this repository and the provided training scripts. I have successfully trained the motion prior using SLAM poses (script 1_train_motion_prior.py). However, I am trying to understand the branch that incorporates egocentric images and HaMeR for guiding the diffusion. I have a few questions regarding this setup:

Dataset Source: I couldn't find a reference to the dataset used for egocentric images. Could you clarify where the dataset is defined or loaded in the code?

Training Script: The only training script I found is for training the motion prior with SLAM poses. Is there a separate script that handles the full training pipeline using egocentric images and HaMeR?

Any guidance on this would be greatly appreciated. Thank you for your time!

[brentyi]

Glad you got training working!!

Images are only used at test time, so we don't have a dataset or script for training. The diffusion model in EgoAllo is only conditioned on SLAM poses.

If you want to follow the details here:

1. We run HaMeR on images from test sequences. For Aria data this is done with 2_run_hamer_on_vrs.py.
2. The inference script (3_aria_inference.py) loads hand information in the form of HaMeR outputs + optionally Aria MPS wrist detections.
   • The loading code is here.
3. The hand inputs are passed to the guidance optimizer. You can find this in the sampling function here.
4. We minimize a mix of prior, 3D, and reprojection losses in do_guidance_optimization().