Add record: Optimizer Tuning + Sliding Window Eval (val_bpb=1.1864)#321

Open

andreanjos wants to merge 76 commits intoopenai:mainfrom

andreanjos:autoresearch/parameter-golf

Commits on Mar 18, 2026

chore: gitignore autoresearch artifacts

autoresearch
and
claude
committed
experiment(train_gpt): replace ReLU² MLP with SwiGLU — hypothesis: gated activation is more expressive at same param count

autoresearch
and
claude
committed
experiment(train_gpt): increase layers 9→11 — hypothesis: more depth uses 16MB budget better with SwiGLU

autoresearch
and
claude
committed
experiment(train_gpt): increase matrix_lr 0.04→0.05 — hypothesis: deeper SwiGLU model benefits from higher Muon LR

autoresearch
and
claude
committed
Revert "experiment(train_gpt): increase matrix_lr 0.04→0.05 — hypothesis: deeper SwiGLU model benefits from higher Muon LR"
autoresearch
committed
experiment(train_gpt): reduce logit_softcap 30→20 — hypothesis: tighter capping sharpens predictions for small vocab

autoresearch
and
claude
committed
experiment(train_gpt): increase model_dim 512→528 — hypothesis: use artifact headroom for wider model

autoresearch
and
claude
committed
Revert "experiment(train_gpt): increase model_dim 512→528 — hypothesis: use artifact headroom for wider model"
autoresearch
committed
experiment(train_gpt): increase warmdown_iters 1200→2400 — hypothesis: longer LR decay improves final convergence

autoresearch
and
claude
committed
experiment(train_gpt): increase muon_backend_steps 5→8 — hypothesis: better orthogonalization improves gradient quality

autoresearch
and
claude
committed
experiment(train_gpt): enable gradient clipping norm=1.0 — hypothesis: stabilizes deeper 11-layer model training

autoresearch
and
claude
committed

Commits on Mar 19, 2026

Commits on Mar 20, 2026

Commits on Mar 21, 2026

Revert "experiment(train_gpt): increase warmdown_iters 10000→12000 — hypothesis: slightly more decay helps 12-layer model"
autoresearch
committed
experiment(train_gpt): reduce tied_embed_init_std 0.005→0.002 — hypothesis: smaller init for deeper 12-layer model

autoresearch
and
claude
committed
Revert "experiment(train_gpt): reduce tied_embed_init_std 0.005→0.002 — hypothesis: smaller init for deeper 12-layer model"
autoresearch
committed
experiment(train_gpt): set layers to 10 — verified on 4xH100: 1.2074 BPB, 15.2MB artifact

autoresearch
and
claude
committed
experiment(train_gpt): 12 layers with thinner SwiGLU (hidden factor 2/3→11/20)

autoresearch
and
claude
committed
submission(train_gpt): 9-layer ReLU² with sliding window eval — 1.1864 BPB on 8xH100

autoresearch
and
claude
committed
Add record: Optimizer Tuning + Sliding Window Eval, val_bpb=1.1864

autoresearch
and
claude
committed