AC model not working in simulation

[ProxyCausal]

Not sure if I really understand the camera angles but I got my cameras in Mujoco to look as close to the figures in the paper as possible

I replaced the photos in the energy_landscape_example notebook with the initial and goal images along with the states

array([ 4.67094602e-01, -1.57964984e-02,  4.64790291e-01, -1.79259928e+02,
       -1.36585258e-03,  9.00057912e+01,  4.00000008e-02])

array([ 4.76420069e-01,  1.31873005e-01,  4.64416030e-01, -1.79251072e+02,
        1.73106157e-01,  9.00084694e+01,  4.00000008e-02])

This is a move of around +13 cm on the y-axis. However, MPC gives 5.0332e-01, -1.1426e-02, 4.4649e-01 as the final state

Maybe I just need to get exactly 37 degrees like the paper says so the rotation error is close to zero

[lassesofus]

Have you tried using a more realistic background?
Constant color may be too out-of-distribution for what the encoders have seen.

I am trying to implement the AC model in a RoboHive environment (also with Mujoco engine), and it kinda works (at least for planning along the y- and z-axis).
See screenshot (x_0: start, x_g: goal, x_k: intermediate steps after planning):

Tracking L2 position error and L1 representation distance over five action planning steps:

You can also do quite a bit of tweaking of the CEM parameters.

I'm running (on an NVIDIA RTX 5090 GPU):
mpc_args={
"rollout": 1,
"samples": 800,
"topk": 10,
"cem_steps": 10,
"momentum_mean": 0.15,
"momentum_mean_gripper": 0.15,
"momentum_std": 0.75,
"momentum_std_gripper": 0.15,
"maxnorm": 0.075,
"verbose": False
}

[ProxyCausal]

hm ok thanks, was going to try using a more realistic environment next (perhaps something from robosuite) but glad to see someone get it working!

yeah that's weird the error doesn't go down closer to 0, did you calibrate the camera angle?

[lassesofus]

Haven't done any calibration yet, but I'm planning on trying out the calibration proposed in the paper. Also, I'm dealing with some suboptimal actuator control explaining at least some of the position error.