Skip to content

Clarification on latent actions ℓ_t vs. physical action chunks a_t in TI2VA/ICL training #109

Description

@DocZbs

Hi LingBot-VA team,

Thank you for releasing LingBot-VA 2.0 and for the detailed paper. I am trying to understand the precise relationship between the learned latent actions and the physical robot-action chunks used in the second-stage video-action model.

My current reading of arXiv:2607.08639v2 is:

  • Sec. 2.1.2 defines a_t as a chunk of raw low-level robot actions aligned with the visual transition z_t -> z_{t+1}.
  • Sec. 2.2.2 learns a latent action l_t = q(z_t, z_{t+1}) from unlabeled video, without action labels. The last paragraph then says that the learned latent-action token is written as a_t = l_t in the second-stage model to match Eq. (5).
  • Sec. 2.3.6 describes TI2VA as future-latent prediction plus inverse-dynamics modeling of the latent actions connecting consecutive states.
  • In contrast, Sec. 2.3.4 describes Eq. (24) using a paired robot trajectory with visual chunks and action chunks, which appears to mean ground-truth physical robot actions.
  • The implementation details in Sec. 4.1.1 describe the action stream as a unified 30-dimensional raw-action space, while Sec. 3.1 says that the general web image/video data are used for T2I/T2V before video-action adaptation.

Could you please clarify the following?

  1. For TI2VA samples, is the action target the tokenizer-produced latent action l_t, the ground-truth physical action chunk a_t, or does this depend on the dataset?
  2. If both latent actions and physical actions are used, how are they aligned? Do they share the same action expert/output head, or are there separate projection/encoder/decoder heads between l_t and the 30D physical-action representation?
  3. Is it correct that a_t in Eq. (24) specifically denotes the ground-truth robot action chunk, rather than l_t, despite the notation a_t = l_t introduced in Sec. 2.2.2?

A short description of the supervision/data flow for web video, robot trajectories, and HCT data would be very helpful.

Thank you very much for your time and for the great work!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions