Monitor lejepa training to completion (225 tasks x 5M steps)

[keyboardAnt]

Context

Lejepa training was submitted across 225 tasks (target: 5M steps each) on the WEXAC HPC cluster. Multiple submission rounds occurred due to failures discovered along the way. This issue tracks the remaining monitoring work until all 225 tasks reach DONE or EXIT.

Current State (as of Feb 23, 2026)

Job Arrays (latest clean submission)

• 674475, 674476, 674477 — 104 jobs total, submitted Feb 23 00:47
• Queues: long-gpu (48h, task indices 1–105), short-gpu (5:45h, indices 106–225)
• Container: /home/projects/dharel/nadavt/newt_roy.sif
• Branch: main

Status Breakdown

• ~120 tasks: Previously completed (from earlier submission rounds)
• ~100 tasks: Running in current batch
• 3 tasks (EXIT): CUDA OOM on shared GPU lgn08 (job 674476 indices 78, 80, 81) — transient, need resubmission
• 1 task (SSUSP): System-suspended, may auto-resume

History of Issues Resolved

1. 46 corrupt checkpoints: .pt files truncated from jobs killed mid-save. Removed all corrupt files, created .removed.txt sidecar documentation (25 tasks fell back to earlier checkpoints, 20 restarted from scratch).
2. Duplicate jobs: Multiple overlapping submission rounds caused all tasks to have HB≥2. Killed everything via bkill -J, waited for heartbeat TTL expiry, clean single resubmit.
3. Pickle cache corruption: runs_cache.pkl unpickling error due to numpy version mismatch (No module named 'numpy._core'). Remove the .pkl file if it recurs.

What Remains

Immediate

• Resubmit the 3 OOM-failed tasks (indices 78, 80, 81 from 674476)

  cd tdmpc2
  python -m runctl --no-wandb submit train --agent lejepa --submit

  This will auto-detect stalled tasks and submit only those.

• Check if the SSUSP job auto-resumes; if not, resubmit it too.

Ongoing

• Monitor until all 225 tasks are DONE or EXIT

  cd tdmpc2
  python -m runctl --no-wandb status
  bjobs -J "newt-train-lejepa*"

• If any new EXIT jobs appear, debug (check outputs/logs/<task>/<run_id>/stderr):

  • OOM: Transient on shared GPUs — resubmit
  • Corrupt checkpoint: Remove .pt, create .removed.txt doc, resubmit
  • Other: Investigate stderr/stdout logs

Completion Criteria

All 225 tasks have status == done (ckpt_step_max >= 5,000,000) or a documented EXIT reason.

How to Check Status

cd tdmpc2
module load miniconda/24.11_environmentally && conda activate newt

# Quick overview
python -m runctl --no-wandb status

# Per-task details
python -m runctl --no-wandb status tasks --format json

# LSF job status
bjobs -J "newt-train-lejepa*"

Notes

• Heartbeat TTL is 30s (HEARTBEAT_TTL_S_DEFAULT in liveness.py)
• If runs_cache.pkl causes errors, delete it: rm outputs/runctl/runs_cache.pkl
• The --no-wandb flag avoids W&B API calls; use it for local-only status checks