feat(action): add RoboCasa mobile-manipulation action-policy dataset and config - #184
Open
hyzhou-nv wants to merge 4 commits into
Open
feat(action): add RoboCasa mobile-manipulation action-policy dataset and config#184hyzhou-nv wants to merge 4 commits into
hyzhou-nv wants to merge 4 commits into
Conversation
Collaborator
|
LGTM |
fwd4
approved these changes
Aug 13, 2026
…and config
Adds RoboCasa support to the action-policy SFT path: a LeRobot dataset
loader, the dataset factory, the Nano post-training config, and the
embodiment registrations.
RoboCasa is a mobile-manipulation benchmark, so the base is not fixed.
The loader exposes a 15-D raw action contract that passes the native
base_motion through unchanged:
[base_motion(4), control_mode(1), eef_pos(3), eef_rot6d(6), gripper(1)]
Observations are the agentview_left | eye_in_hand composite at full
resolution, with EEF proprioception as a prepended clean condition token.
All changes are additive; no existing line is modified.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Moves the RoboCasa evaluator out of the cookbook and into the framework,
next to the LIBERO one, per review feedback: the DROID and LIBERO
cookbooks ship training code only.
cosmos_framework/simulation/robocasa/
closed_loop_eval.py drives the simulator against the policy server
eval_utils.py action decoders, HTTP call, generated-video decode
Only the empty namespace packages are imported from cosmos_framework, so
PYTHONPATH=. suffices and the framework need not be installed in the
robosuite/robocasa venv -- the two cannot share one environment.
Three defects are fixed relative to the cookbook version:
* Camera observations are flipped upright before composing. MuJoCo renders
bottom-up while the recorded LeRobot videos are upright, so the policy was
being shown an inverted world.
* The language annotation is read after env.reset() rather than before.
get_ep_meta() describes the scene currently loaded, so reading it first
returned the previous episode's metadata, and nothing at all on the first
rollout. For tasks whose prompt names the sampled object, every rollout
was conditioned on the previous scene's object.
* Rollouts per task is a parameter rather than hardcoded at 50.
Verified end-to-end on GB200: imports resolve in the simulator venv without
pulling torch, and rollouts complete with each episode's prompt matching its
own scene.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
hyzhou-nv
force-pushed
the
pr/robocasa-action-policy
branch
from
August 14, 2026 16:09
270adab to
1892cac
Compare
The RoboCasa recipe trains with `use_state=True`: the current end-effector
pose is prepended to the action sequence as a clean conditioning frame at
index 0, so the sequence is `chunk_length + 1` long and the model plans
relative to where the arm actually is. The policy server had no counterpart,
so such a checkpoint was served with a `chunk_length`-long all-zero action
tensor and the `"state"` field in the request was silently ignored.
Measured on RoboCasa target/atomic at iter 16000, 18 tasks x 20 rollouts,
official-test protocol, same weights either way:
without the state token 226/360 = 63%
with the state token 287/360 = 80%
Requests that do not carry "state" are unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
hyzhou-nv
force-pushed
the
pr/robocasa-action-policy
branch
from
August 18, 2026 03:31
95a5868 to
e0e4e6c
Compare
Moves the dataset preparation out of the cookbook, per review, so the recipe folder holds only training files. Sits alongside the other convert_* scripts. RoboCasa publishes LeRobot v2.1 while the pinned lerobot is v3.0-only and rejects that layout, so this is a prerequisite for training rather than optional setup. It works on copies, is idempotent, and converts every task it finds under SRC_ROOT, so one script covers target/atomic, pretrain/atomic and pretrain/composite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds RoboCasa support to the action-policy SFT path: a LeRobot dataset loader, the dataset factory, the Nano post-training config, and the embodiment registrations.
RoboCasa is a mobile-manipulation benchmark, so the base is not fixed. The loader exposes a 15-D raw action contract that passes the native base_motion through unchanged:
Observations are the agentview_left | eye_in_hand composite at full resolution, with EEF proprioception as a prepended clean condition token.
All changes are additive; no existing line is modified.