Reproducing the Fig. 5 / calibration hallucination-predictor correlations: what exactly was evaluated?

[sai-prasanna]

Thanks for releasing the code, checkpoints, and the interactive plot. It's been great to work with.

We're trying to reproduce the u_r / u_f / u_s vs ΔPSNR rank correlations from Fig. 5, i.e. the per-sequence evaluation exposed in the interactive plot's calibration.js (n_total = 8976, ρ = -0.808 / -0.785 / -0.796 for u_r_norm / u_f_norm / u_s_norm). Using the released base checkpoint and the code in this repo we get ρ = -0.61 / -0.65 / -0.67 for the same three predictors (a single predictor estimate from the clean context at t = ctx_length), so we seem to be missing something in the evaluation protocol rather than the method.

Would you be able to share the evaluation script that produced these Fig. 5 numbers (the per-sequence values in calibration.js)? It doesn't look like it's in the repo, and it would let us line up our reproduction exactly against yours.

What we believe we've matched already (please correct any of these):

• Checkpoint: released base (tokenizer.pt + dynamics.pt).
• Predictors: URNormScorer (u_r, motion = RMS(z_pred - z_prev)) and CrossSeedScorer (u_s) from uncertainty.py; u_f = tail-half mean of per-substep RMS Δx̂₁ from interactive.py.
• Rollout: sample_autoregressive_packed_sequence, open-loop, ctx_length=8, horizon=16, seq_len=24.
• ΔPSNR: single MSE over the whole horizon to dB, psnr_pred - psnr_floor, floor = repeat last context frame (as in log_dynamics_eval_wandb).

If the script isn't easy to share, the specific things we're unsure about:

1. Eval sequence selection (our main unknown). n_total = 8976 = 187 tasks × 48 sequences. How were the 48 sequences per task drawn: which trajectories, and how is each 24-frame window chosen within a trajectory (random / strided / first-N / filtered by quality or motion)?

2. Where is each predictor evaluated? Is the reported predictor a single estimate from the clean context at the boundary (t = ctx_length), or averaged over multiple positions along the open-loop rollout?

3. tau_ctx. The eval driver defaults tau_ctx=0.0 while sample_predictions_for_actions defaults 0.1. Which value was used (a) for the ΔPSNR rollout and (b) for the predictor sampling for these numbers?

4. u_s seeds. CrossSeedScorer takes variance over n_samples (default 2). How many seeds were used for the reported u_s?

5. ΔPSNR reduction. Was ΔPSNR computed exactly as in log_dynamics_eval_wandb (one MSE over all horizon frames to dB), or per-frame PSNR averaged over the horizon, or with any clipping? (Your y values range down to ~-93 dB, so presumably unclipped for the correlation even though the plot displays [-5, 15].)

Thanks!