infra setup / workflow

[rosikand]

I am building my own custom workflow to test and iterate on ideas quickly. Ideally, we can fire off elastic runs serverless-ly.

the workflow is: implement new method in train_gpt.py, run the run.py command --> launches to modal with wandb logging --> results saved to my own local leaderboard state.

Notes:

• Need to make sure we are auto-constraining the limits with modal. 10min train + 10min eval, 16mb memory footprint.

[rosikand]

Experiments

Keep a directory of variants:

  experiments/                                                                                                                                                                                                    
  ├── baseline.py
  ├── depth_recurrence.py                                                                                                                                                                                         
  ├── bitnet_quantized.py
  └── sliding_window_v2.py

Then point run.py at whichever one:

python infra/run.py --script experiments/depth_recurrence.py --name "depth_recurrence_v1"

And for batch firing:

python infra/run.py --script experiments/*.py --gpus 1

train_gpt.py stays as the upstream reference. Your work lives in experiments/.

[rosikand]

The workflow would be:

1. Edit train_gpt.py locally (new architecture, loss fn, whatever)
2. Run: python infra/run.py --name "depth_recurrence_v1" --gpus 1
3. Modal uploads YOUR current train_gpt.py to an H100, runs it, streams logs
4. Results auto-log to W&B + append to local leaderboard

For sweeps, you'd still be running the same modified script but varying env vars:

python infra/sweep.py --name "recurrence_lr_sweep" --gpus 1 \
--vary LEARNING_RATE=0.001,0.003,0.01 \
--vary NUM_LAYERS=6,9,12

So it's your script, their GPUs. Modal just handles the "ship this file to a machine with H100s and run it" part.

Structure:

infra/
 ├── modal_app.py       # Modal function: uploads train_gpt.py, runs on H100, logs to W&B
 ├── run.py             # single run: python infra/run.py --name "my_idea" --gpus 1                                                                                                                              
 ├── sweep.py           # fan out: vary env vars across N parallel runs                                                                                                                                          
 ├── leaderboard.py     # parse results, update leaderboard files                                                                                                                                                
 ├── leaderboard.csv    # machine-readable experiment log                                                                                                                                                        
 └── leaderboard.md     # human-readable: runs, techniques, notes, progression  

[rosikand]

Files created:

• infra/config.py — shared constants (Modal volume, W&B project, default env vars, cost rates)
• infra/log_parser.py — regex parser for train_gpt.py structured output (tested, all patterns work)
• infra/modal_app.py — Modal app with GPU-dispatched functions (1/2/4/8x H100), W&B logging in the wrapper, persistent volume for dataset
• infra/leaderboard.py — append-only CSV + auto-regenerated LEADERBOARD.md
• infra/run.py — single run CLI with --dry-run, --env, --ensure-data
• infra/sweep.py — parallel sweep CLI with cartesian product, cost estimates, confirmation prompt
• experiments/baseline.py — copy of train_gpt.py as starting point
• LEADERBOARD.md — empty leaderboard ready to go

Next steps to go live:

1. pip install modal wandb
2. modal setup (one-time auth)
3. modal secret create wandb-secret WANDB_API_KEY=<your_key>
4. python infra/run.py --ensure-data (downloads dataset to Modal volume, ~5 min)
5. python infra/run.py --script experiments/baseline.py --name "baseline" --gpus 1 (first real run)