Hello,
I just encountered an issue inside the training script.
Every training run creates two sibling experiment directories under logs/:
logs/project/XXX/ ← weights, config, etc.
logs/project/XXX/ ← renderings_training/ (empty)
get_timestamp() is called twice independently:
train_agent.py:182 — generates the timestamp used for logging, config, checkpoints
base_task.py:82 — generates a second timestamp a few seconds later, used for the simulator/video recorder's experiment_dir
Since the environment setup (IsaacGym init, wandb init, etc.) takes a few seconds between the two calls, the timestamps differ, creating two directories.
The video recorder writes to the second directory (renderings_training/), but when wandb is enabled the local video files are cleaned up after upload, leaving it empty.
I have created a commit that should solve the issue, should I open a PR?
Thanks for the help!
Hello,
I just encountered an issue inside the training script.
Every training run creates two sibling experiment directories under
logs/:logs/project/XXX/ ← weights, config, etc.
logs/project/XXX/ ← renderings_training/ (empty)
get_timestamp()is called twice independently:train_agent.py:182— generates the timestamp used for logging, config, checkpointsbase_task.py:82— generates a second timestamp a few seconds later, used for the simulator/video recorder'sexperiment_dirSince the environment setup (IsaacGym init, wandb init, etc.) takes a few seconds between the two calls, the timestamps differ, creating two directories.
The video recorder writes to the second directory (
renderings_training/), but when wandb is enabled the local video files are cleaned up after upload, leaving it empty.I have created a commit that should solve the issue, should I open a PR?
Thanks for the help!