Hi LingBot-VA team,
Thank you for releasing LingBot-VA 2.0 and for the detailed paper. I am trying to understand the precise relationship between the learned latent actions and the physical robot-action chunks used in the second-stage video-action model.
My current reading of arXiv:2607.08639v2 is:
- Sec. 2.1.2 defines
a_t as a chunk of raw low-level robot actions aligned with the visual transition z_t -> z_{t+1}.
- Sec. 2.2.2 learns a latent action
l_t = q(z_t, z_{t+1}) from unlabeled video, without action labels. The last paragraph then says that the learned latent-action token is written as a_t = l_t in the second-stage model to match Eq. (5).
- Sec. 2.3.6 describes TI2VA as future-latent prediction plus inverse-dynamics modeling of the latent actions connecting consecutive states.
- In contrast, Sec. 2.3.4 describes Eq. (24) using a paired robot trajectory with visual chunks and action chunks, which appears to mean ground-truth physical robot actions.
- The implementation details in Sec. 4.1.1 describe the action stream as a unified 30-dimensional raw-action space, while Sec. 3.1 says that the general web image/video data are used for T2I/T2V before video-action adaptation.
Could you please clarify the following?
- For TI2VA samples, is the action target the tokenizer-produced latent action
l_t, the ground-truth physical action chunk a_t, or does this depend on the dataset?
- If both latent actions and physical actions are used, how are they aligned? Do they share the same action expert/output head, or are there separate projection/encoder/decoder heads between
l_t and the 30D physical-action representation?
- Is it correct that
a_t in Eq. (24) specifically denotes the ground-truth robot action chunk, rather than l_t, despite the notation a_t = l_t introduced in Sec. 2.2.2?
A short description of the supervision/data flow for web video, robot trajectories, and HCT data would be very helpful.
Thank you very much for your time and for the great work!
Hi LingBot-VA team,
Thank you for releasing LingBot-VA 2.0 and for the detailed paper. I am trying to understand the precise relationship between the learned latent actions and the physical robot-action chunks used in the second-stage video-action model.
My current reading of arXiv:2607.08639v2 is:
a_tas a chunk of raw low-level robot actions aligned with the visual transitionz_t -> z_{t+1}.l_t = q(z_t, z_{t+1})from unlabeled video, without action labels. The last paragraph then says that the learned latent-action token is written asa_t = l_tin the second-stage model to match Eq. (5).Could you please clarify the following?
l_t, the ground-truth physical action chunka_t, or does this depend on the dataset?l_tand the 30D physical-action representation?a_tin Eq. (24) specifically denotes the ground-truth robot action chunk, rather thanl_t, despite the notationa_t = l_tintroduced in Sec. 2.2.2?A short description of the supervision/data flow for web video, robot trajectories, and HCT data would be very helpful.
Thank you very much for your time and for the great work!