Skip to content

Training creates two experiment directories with different timestamps #44

@maedmatt

Description

@maedmatt

Hello,

I just encountered an issue inside the training script.

Every training run creates two sibling experiment directories under logs/:

logs/project/XXX/ ← weights, config, etc.
logs/project/XXX/ ← renderings_training/ (empty)

get_timestamp() is called twice independently:

  1. train_agent.py:182 — generates the timestamp used for logging, config, checkpoints
  2. base_task.py:82 — generates a second timestamp a few seconds later, used for the simulator/video recorder's experiment_dir

Since the environment setup (IsaacGym init, wandb init, etc.) takes a few seconds between the two calls, the timestamps differ, creating two directories.

The video recorder writes to the second directory (renderings_training/), but when wandb is enabled the local video files are cleaned up after upload, leaving it empty.

I have created a commit that should solve the issue, should I open a PR?

Thanks for the help!

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions