Unable to reproduce Cosmos3-Nano-Policy-DROID results on RoboLab-120

[Antu3heng]

Hi, thank you for releasing the code and checkpoints for Cosmos3.

I am trying to reproduce the Cosmos3-Nano-Policy-DROID results on the RoboLab-120 benchmark. However, the success rates I obtained are substantially lower than those reported in Table 19 of the paper, especially for the Default and Specific language settings.

Results

Results reported in the paper

Difficulty	Vague	Default	Specific
Overall	20.6	36.8	39.7
Simple	23.3	40.6	42.0
Moderate	23.3	35.4	40.3
Complex	4.1	25.3	29.4

My reproduced results

4 denoising steps

Difficulty	Vague	Default	Specific
Overall	19.4	26.5	27.9
Simple	22.2	29.7	28.4
Moderate	21.5	23.8	27.9
Complex	4.1	20.6	25.9

10 denoising steps

Difficulty	Vague	Default	Specific
Overall	17.8	26.5	28.9
Simple	21.1	29.8	29.4
Moderate	19.0	24.9	29.2
Complex	2.9	17.6	26.5

With 4 denoising steps which is setting of the released code, the reproduced Overall scores are:

• Vague: 19.4 vs. 20.6 reported
• Default: 26.5 vs. 36.8 reported
• Specific: 27.9 vs. 39.7 reported

The Vague setting is relatively close to the paper, but the Default and Specific settings are approximately 10–14 percentage points lower for most Simple and Moderate task categories. Increasing the number of action steps from 4 to 10 does not close the gap.

Reproduction setup

• Model/checkpoint: Cosmos3-Nano-Policy-DROID
• Benchmark: RoboLab-120
• Denoising steps tested: 4 and 10
• Number of episodes per task: 10
• Hardware: H20 for COSMOS policy and RTX 5090 for RoboLab-120

I followed the released evaluation instructions and used the provided off-the-shelf DROID checkpoint without additional fine-tuning.

Questions

Could you please clarify the following?

1. Is there a specific evaluation configuration for RoboLab-120 that is not included in the default config?
2. What action chunk size / number of denoising steps / denoising schedule shift was used for the reported results?
3. Were any additional preprocessing, prompt templates, language-instruction transformations, or normalization steps applied?
4. How many evaluation episodes and random seeds were used for each task?
5. Are there any known differences between the released checkpoint/evaluation pipeline and the internal setup used for the paper?

Any guidance or a complete evaluation command/configuration for reproducing Table 19 would be greatly appreciated. Thank you!

[mli0603]

@Antu3heng this looks like a bug as the SR drops so much. You should be able to reproduce the results with 4 steps, no need for 10 steps. What hardware are you using? I suspect it is a hardware + issac version issue. Have you checked the rendered images / videos of Robolab?

1. Is there a specific evaluation configuration for RoboLab-120 that is not included in the default config?
   No
2. What action chunk size / number of denoising steps / denoising schedule shift was used for the reported results?
   The same one as released
3. Were any additional preprocessing, prompt templates, language-instruction transformations, or normalization steps applied?
   No
4. How many evaluation episodes and random seeds were used for each task?
   @fwd4 do you remember this?
5. Are there any known differences between the released checkpoint/evaluation pipeline and the internal setup used for the paper?
   No. And we are know that many people in the community was able to reproduce the results within the error range (i.e., around 40% success rate).

[Antu3heng]

@mli0603 Thank you for the response. It is helpful to know that the reported results should be reproducible using 4 action steps and the released configuration.

My hardware and software environment for the RoboLab-120 evaluation is as follows:

• GPU: NVIDIA RTX 5090
• GPU driver: 570.211.01
• CUDA: 12.8
• Isaac Sim: 5.0.0-rc.45
• Isaac Lab: 2.2.0
• RoboLab commit: 7d45d74904eade3b578a8eb1f2f9f89bc3d40326

I checked the rendered RoboLab videos and did not notice any obvious rendering or simulation issues. The camera observations, robot poses, object positions, and textures appear normal.

I have attached one representative failure video below:

Put_the_bagels_on_the_plate_0_env0_viewport.mp4

In many failed episodes, I observed that the robot approaches the correct object but fails to grasp it or fails to complete the final placement.

Thanks again for your help!

[mli0603]

@Antu3heng Are you using single view?

[fwd4]

@mli0603 @Antu3heng seed we use for evaluation is random.

Also I noticed the table texture is missing in your videos? The table looks in our evaluation:

sim_wrist.mp4

[Antu3heng]

@mli0603 @fwd4 Thank you again for your help. I am using the multi-view setting. Regarding the difference in the table texture, could it be caused by randomized environment or texture selection during evaluation?

I also found an issue in my evaluation setup: I had accidentally set the input FPS to 20 FPS instead of the expected 15 FPS. After restoring it to 15 FPS and rerunning the evaluation with 4 denoising steps, the results improved substantially.

Updated results

Difficulty	Vague	Default	Specific
Overall	22.4	31.1	35.3
Simple	24.7	34.8	36.9
Moderate	25.1	29.0	35.6
Complex	7.6	21.8	28.8

For comparison, Table 19 of the paper reports:

Difficulty	Vague	Default	Specific
Overall	20.6	36.8	39.7
Simple	23.3	40.6	42.0
Moderate	23.3	35.4	40.3
Complex	4.1	25.3	29.4

The corrected overall results are now much closer to the reported results:

• Vague: 22.4 vs. 20.6
• Default: 31.1 vs. 36.8
• Specific: 35.3 vs. 39.7

This indicates that the incorrect FPS setting was the main cause of the large discrepancy I reported previously, rather than an apparent hardware or rendering issue. I apologize for the confusion, and thank you for pointing me in the right direction.

A smaller gap remains in the Default and Specific settings. Could you please confirm whether a difference of approximately 4–6 percentage points is within the expected evaluation variance?

However, in my experiments, the success rate for put_bagels_on_the_plate is 0.0, whereas the task succeeds in your video. There may still be an issue in my evaluation setup that I have not identified.

Thanks again for your help!

[fwd4]

Could you please confirm whether a difference of approximately 4–6 percentage points is within the expected evaluation variance?

No, 1-2pp of success rate variance is observed on our side.

Regarding the difference in the table texture, could it be caused by randomized environment or texture selection during evaluation?

AFAIK, robolab uses fixed enviroments (w.r.t objection/robot init positions), the texture bug might relate to simulation. Did you find similar texture issues in other tasks?

[jingjingchengcai]

@fwd4 @Antu3heng
Weird—I used the default setup and also got a score of about 29 on the simple benchmark with the default prompt.

My FPS is 15, and I used the default multi-view setup. Is there anything I might be missing to reproduce those results?