openai · seekerPrice · Apr 14, 2026
diff --git a/records/track_non_record_16mb/2026-04-14_MLX_Tuned_Hparams_1.5096/README.md b/records/track_non_record_16mb/2026-04-14_MLX_Tuned_Hparams_1.5096/README.md
@@ -0,0 +1,96 @@
+# Tuned Hyperparameters for MLX Baseline — 1.5096 BPB Locally
+
+**Author:** @seekerPrice
+**Date:** 2026-04-14
+**Track:** non_record_16mb (H100 validation pending)
+**Hardware:** Apple M5 MacBook Pro (MLX framework)
+
+## TL;DR
+
+A/B comparison at **same model, same training config, different hyperparameters**:
+
+| Experiment | Matrix LR | Muon Momentum | QK-Gain | val_bpb |
+|-----------|-----------|---------------|---------|---------|
+| EXP-042 (SOTA defaults) | 0.022 | 0.99 | 5.25 | 1.5596 |
+| **EXP-048 (tuned)** | **0.02** | **0.95** | **4.0** | **1.5096** |
+| | | | **Δ:** | **-0.0500** |
+
+**Pure hyperparameter tuning gave -0.05 BPB** at 5000-step MLX training scale.
+
+## Changes
+
+Only 4 hyperparameters changed. Same architecture (11L × 512d × 4xMLP, depth recurrence L3,4,5, parallel residuals L7+, SP4096 casefold tokenizer, Muon + AdamW split optimizer).
+
+```diff
+-matrix_lr = 0.022
++matrix_lr = 0.02
+
+-muon_momentum = 0.99
++muon_momentum = 0.95
+
+-muon_momentum_warmup_start = 0.95
++muon_momentum_warmup_start = 0.90
+
+-qk_gain_init = 5.25
++qk_gain_init = 4.0
+```
+
+## Why These Values
+
+**Starting from SOTA defaults** (tuned for H100 large-batch training), we hypothesized they might be too aggressive for our small-batch MLX runs:
+
+- **Matrix LR 0.022 → 0.02**: less aggressive update magnitude at smaller batch (8K tokens vs SOTA's 524K)
+- **Muon momentum 0.99 → 0.95**: less backward-looking; helps in small-batch noisy gradient regime
+- **Muon momentum warmup 0.95 → 0.90**: slower warmup reduces early training spikes
+- **QK-Gain 5.25 → 4.0**: softer attention. Pairs well with Partial RoPE (only 16/64 dims rotated) — too-sharp attention overreacts to the non-rotated content dimensions
+
+## Experimental Setup
+
+**Training:** 5000 steps, 8K tokens/step = 40M total tokens (vs SOTA H100: 2.4B)
+**Validation:** Full FineWeb val split (47.7M tokens, 524K batch)
+**Model:** 33.8M params, ~12.65 MB artifact after int6+Brotli compression
+
+**Environment variables (EXP-048 winning config):**
+```bash
+export MATRIX_LR=0.02
+export MUON_MOMENTUM=0.95
+export MUON_MOMENTUM_WARMUP_START=0.90
+export QK_GAIN_INIT=4.0
+# Other (unchanged from SOTA):
+export TIED_EMBED_LR=0.03
+export SCALAR_LR=0.02
+export GRAD_CLIP_NORM=0.3
+export MUON_MOMENTUM_WARMUP_STEPS=60
+```
+
+## Key Caveat
+
+**This is a LOCAL MLX result at 40M tokens — not a H100 competition submission.**
+
+The SOTA leaderboard results are at 2.4B tokens on 8×H100. Our 40M-token result isn't directly comparable. However, the **A/B improvement within our framework** (-0.05 BPB from hyperparameter tuning alone) should transfer to larger scales — tuned hyperparameters are generally scale-stable for small deltas like these.
+
+**Prediction:** At H100 scale, these tuned values should give ~0.01-0.02 BPB improvement over SOTA-default hyperparameters, all else equal.
+
+## Why Share This
+
+Small improvements accumulate. If these tuned hyperparameters give even -0.005 BPB at H100 scale, that's meaningful against the 1.0810 leaderboard SOTA. Sharing empirical evidence helps the community.
+
+## Methodology Notes
+
+**3-AI collaboration** (Claude + Gemini + Codex) independently recommended these exact hyperparameters based on theoretical analysis (Muon at large momentum is unstable with small batch; QK-Gain 5.25 over-sharpens with partial RoPE). We then validated empirically.
+
+## Status
+
+- [x] Local MLX A/B test (5000 steps) — EXP-042 vs EXP-048
+- [x] Documented in project findings
+- [ ] H100 3-seed validation (pending compute credits)
+- [ ] Combined with SOTA architecture stack on H100
+
+## Related PRs
+
+- #1595 (open): Previous non-record submission (3x MLP + QAT) — superseded by this result
+- Applying for H100 credits at: https://jobs.ashbyhq.com/openai/form/open-ai-challenge-parameter-golf
+
+## Attribution
+
+Baseline SOTA hyperparameters sourced from PR #1394 (clarkkev), #1437 (dexhunter), #1412 (Robby955), #1445 (X-Abhishek-X). Our contribution is the specific re-tuning for the 8K-batch MLX regime and empirical validation.
diff --git a/records/track_non_record_16mb/2026-04-14_MLX_Tuned_Hparams_1.5096/exp042_train.log b/records/track_non_record_16mb/2026-04-14_MLX_Tuned_Hparams_1.5096/exp042_train.log
@@ -0,0 +1,100 @@
+val_bpb:enabled tokenizer_kind=sentencepiece tokenizer_path=./data/tokenizers/fineweb_4096_bpe.model
+warmup_step:1/10
+warmup_step:2/10
+warmup_step:3/10
+warmup_step:4/10
+warmup_step:5/10
+warmup_step:6/10
+warmup_step:7/10
+warmup_step:8/10
+warmup_step:9/10
+warmup_step:10/10
+step:10/5000 train_loss:6.9618 train_time:8442ms step_avg:844.21ms tok_s:9470
+step:200/5000 train_loss:4.7006 train_time:167210ms step_avg:836.05ms tok_s:9846
+step:400/5000 train_loss:4.5413 train_time:334295ms step_avg:835.74ms tok_s:9960
+step:600/5000 train_loss:4.5037 train_time:501564ms step_avg:835.94ms tok_s:9409
+step:800/5000 train_loss:4.1423 train_time:670228ms step_avg:837.79ms tok_s:9138
+step:1000/5000 train_loss:4.1259 train_time:841046ms step_avg:841.05ms tok_s:9672
+step:1200/5000 train_loss:4.0284 train_time:1008541ms step_avg:840.45ms tok_s:9573
+step:1400/5000 train_loss:3.9586 train_time:1175981ms step_avg:839.99ms tok_s:10052
+step:1600/5000 train_loss:4.0497 train_time:1336270ms step_avg:835.17ms tok_s:10720
+step:1800/5000 train_loss:3.7127 train_time:1489182ms step_avg:827.32ms tok_s:10721
+step:2000/5000 train_loss:3.5694 train_time:1641897ms step_avg:820.95ms tok_s:10735
+step:2200/5000 train_loss:3.7285 train_time:1794625ms step_avg:815.74ms tok_s:10737
+step:2400/5000 train_loss:3.4214 train_time:1947544ms step_avg:811.48ms tok_s:10720
+step:2600/5000 train_loss:3.6573 train_time:2100417ms step_avg:807.85ms tok_s:10719
+step:2800/5000 train_loss:3.4875 train_time:2253109ms step_avg:804.68ms tok_s:10746
+step:3000/5000 train_loss:3.2726 train_time:2405150ms step_avg:801.72ms tok_s:10698
+step:3200/5000 train_loss:3.4832 train_time:2557052ms step_avg:799.08ms tok_s:10825
+step:3400/5000 train_loss:3.5935 train_time:2708819ms step_avg:796.71ms tok_s:10819
+step:3600/5000 train_loss:3.1340 train_time:2860581ms step_avg:794.61ms tok_s:10815
+step:3800/5000 train_loss:3.4380 train_time:3012279ms step_avg:792.71ms tok_s:10819
+step:4000/5000 train_loss:3.3349 train_time:3163927ms step_avg:790.98ms tok_s:10822
+step:4200/5000 train_loss:3.3645 train_time:3315670ms step_avg:789.45ms tok_s:10802
+step:4400/5000 train_loss:2.8631 train_time:3467366ms step_avg:788.04ms tok_s:10830
+step:4600/5000 train_loss:3.1664 train_time:3618973ms step_avg:786.73ms tok_s:10803
+step:4800/5000 train_loss:3.2211 train_time:3770557ms step_avg:785.53ms tok_s:10796
+step:5000/5000 train_loss:3.2052 train_time:3922232ms step_avg:784.45ms tok_s:10807
+step:5000/5000 val_loss:3.1687 val_bpb:1.5596 train_time:3922233ms step_avg:784.45ms
+    val_loss, val_bpb = eval_val(
+    muon_momentum_warmup_steps: int = int(os.environ.get("MUON_MOMENTUM_WARMUP_STEPS", 500))
+        if self.args.muon_momentum_warmup_steps:
+            t = min(step / self.args.muon_momentum_warmup_steps, 1.0)
+    # The shard directory and tokenizer are coupled: val_bpb is only meaningful if we
+    # - val_bpb: tokenizer-agnostic compression metric used by the challenge
+    val_bpb = bits_per_token * (total_tokens / total_bytes)
+    return val_loss, val_bpb
+        f"model_params:{n_params} vocab_size:{args.vocab_size} layers:{args.num_layers} "
+        f"iterations:{args.iterations} train_batch_tokens:{args.train_batch_tokens} grad_accum_steps:{args.grad_accum_steps} "
+        f"warmup_steps:{args.warmup_steps} max_wallclock_seconds:{args.max_wallclock_seconds:.3f}"
+    log(f"val_bpb:enabled tokenizer_kind=sentencepiece tokenizer_path={args.tokenizer_path}")
+    if args.warmup_steps > 0:
+        for warmup_step in range(args.warmup_steps):
+            if args.warmup_steps <= 20 or (warmup_step + 1) % 10 == 0 or warmup_step + 1 == args.warmup_steps:
+                log(f"warmup_step:{warmup_step + 1}/{args.warmup_steps}")
+            val_loss, val_bpb = eval_val(
+                    f"step:{step}/{args.iterations} val_loss:{val_loss:.4f} val_bpb:{val_bpb:.4f} "
+            if stop_after_step is not None and step < args.iterations:
+    q_val_loss, q_val_bpb = eval_val(
+    log(f"final_int8_zlib_roundtrip val_loss:{q_val_loss:.4f} val_bpb:{q_val_bpb:.4f} eval_time:{q_eval_ms:.0f}ms")
+    log(f"final_int8_zlib_roundtrip_exact val_loss:{q_val_loss:.8f} val_bpb:{q_val_bpb:.8f}")
+model_params:33842864 vocab_size:4096 layers:11 dim:512 heads:8 kv_heads:4 seq_len:1024 tie_embeddings:True
+iterations:5000 train_batch_tokens:8192 grad_accum_steps:2 microbatch_tokens:4096 microbatch_batch_size:4 val_batch_size:524288 warmup_steps:10 max_wallclock_seconds:0.000
+val_bpb:enabled tokenizer_kind=sentencepiece tokenizer_path=./data/tokenizers/fineweb_4096_bpe.model
+warmup_step:1/10
+warmup_step:2/10
+warmup_step:3/10
+warmup_step:4/10
+warmup_step:5/10
+warmup_step:6/10
+warmup_step:7/10
+warmup_step:8/10
+warmup_step:9/10
+warmup_step:10/10
+step:10/5000 train_loss:6.9618 train_time:8442ms step_avg:844.21ms tok_s:9470
+step:200/5000 train_loss:4.7006 train_time:167210ms step_avg:836.05ms tok_s:9846
+step:400/5000 train_loss:4.5413 train_time:334295ms step_avg:835.74ms tok_s:9960
+step:600/5000 train_loss:4.5037 train_time:501564ms step_avg:835.94ms tok_s:9409
+step:800/5000 train_loss:4.1423 train_time:670228ms step_avg:837.79ms tok_s:9138
+step:1000/5000 train_loss:4.1259 train_time:841046ms step_avg:841.05ms tok_s:9672
+step:1200/5000 train_loss:4.0284 train_time:1008541ms step_avg:840.45ms tok_s:9573
+step:1400/5000 train_loss:3.9586 train_time:1175981ms step_avg:839.99ms tok_s:10052
+step:1600/5000 train_loss:4.0497 train_time:1336270ms step_avg:835.17ms tok_s:10720
+step:1800/5000 train_loss:3.7127 train_time:1489182ms step_avg:827.32ms tok_s:10721
+step:2000/5000 train_loss:3.5694 train_time:1641897ms step_avg:820.95ms tok_s:10735
+step:2200/5000 train_loss:3.7285 train_time:1794625ms step_avg:815.74ms tok_s:10737
+step:2400/5000 train_loss:3.4214 train_time:1947544ms step_avg:811.48ms tok_s:10720
+step:2600/5000 train_loss:3.6573 train_time:2100417ms step_avg:807.85ms tok_s:10719
+step:2800/5000 train_loss:3.4875 train_time:2253109ms step_avg:804.68ms tok_s:10746
+step:3000/5000 train_loss:3.2726 train_time:2405150ms step_avg:801.72ms tok_s:10698
+step:3200/5000 train_loss:3.4832 train_time:2557052ms step_avg:799.08ms tok_s:10825
+step:3400/5000 train_loss:3.5935 train_time:2708819ms step_avg:796.71ms tok_s:10819
+step:3600/5000 train_loss:3.1340 train_time:2860581ms step_avg:794.61ms tok_s:10815
+step:3800/5000 train_loss:3.4380 train_time:3012279ms step_avg:792.71ms tok_s:10819
+step:4000/5000 train_loss:3.3349 train_time:3163927ms step_avg:790.98ms tok_s:10822
+step:4200/5000 train_loss:3.3645 train_time:3315670ms step_avg:789.45ms tok_s:10802
+step:4400/5000 train_loss:2.8631 train_time:3467366ms step_avg:788.04ms tok_s:10830
+step:4600/5000 train_loss:3.1664 train_time:3618973ms step_avg:786.73ms tok_s:10803
+step:4800/5000 train_loss:3.2211 train_time:3770557ms step_avg:785.53ms tok_s:10796
+step:5000/5000 train_loss:3.2052 train_time:3922232ms step_avg:784.45ms tok_s:10807
+step:5000/5000 val_loss:3.1687 val_bpb:1.5596 train_time:3922233ms step_avg:784.45ms
diff --git a/records/track_non_record_16mb/2026-04-14_MLX_Tuned_Hparams_1.5096/exp048_train.log b/records/track_non_record_16mb/2026-04-14_MLX_Tuned_Hparams_1.5096/exp048_train.log
@@ -0,0 +1,100 @@
+iterations:5000 train_batch_tokens:8192 grad_accum_steps:2 microbatch_tokens:4096 microbatch_batch_size:4 val_batch_size:524288 warmup_steps:10 max_wallclock_seconds:0.000
+val_bpb:enabled tokenizer_kind=sentencepiece tokenizer_path=./data/tokenizers/fineweb_4096_bpe.model
+warmup_step:1/10
+warmup_step:2/10
+warmup_step:3/10
+warmup_step:4/10
+warmup_step:5/10
+warmup_step:6/10
+warmup_step:7/10
+warmup_step:8/10
+warmup_step:9/10
+warmup_step:10/10
+step:10/5000 train_loss:7.0435 train_time:8189ms step_avg:818.90ms tok_s:10544
+step:200/5000 train_loss:4.5148 train_time:153865ms step_avg:769.33ms tok_s:10777
+step:400/5000 train_loss:4.3341 train_time:306493ms step_avg:766.23ms tok_s:10707
+step:600/5000 train_loss:4.3689 train_time:459261ms step_avg:765.43ms tok_s:10775
+step:800/5000 train_loss:3.9403 train_time:612662ms step_avg:765.83ms tok_s:10764
+step:1000/5000 train_loss:3.7845 train_time:765035ms step_avg:765.04ms tok_s:10750
+step:1200/5000 train_loss:3.7428 train_time:917720ms step_avg:764.77ms tok_s:10763
+step:1400/5000 train_loss:3.4893 train_time:1070595ms step_avg:764.71ms tok_s:10602
+step:1600/5000 train_loss:3.7819 train_time:1223754ms step_avg:764.85ms tok_s:10798
+step:1800/5000 train_loss:3.5227 train_time:1376010ms step_avg:764.45ms tok_s:10763
+step:2000/5000 train_loss:3.4185 train_time:1528915ms step_avg:764.46ms tok_s:10786
+step:2200/5000 train_loss:3.5589 train_time:1683208ms step_avg:765.09ms tok_s:10774
+step:2400/5000 train_loss:3.2930 train_time:1835796ms step_avg:764.91ms tok_s:10763
+step:2600/5000 train_loss:3.5559 train_time:1988350ms step_avg:764.75ms tok_s:10779
+step:2800/5000 train_loss:3.4060 train_time:2140679ms step_avg:764.53ms tok_s:10779
+step:3000/5000 train_loss:3.1498 train_time:2293002ms step_avg:764.33ms tok_s:10796
+step:3200/5000 train_loss:3.3820 train_time:2448866ms step_avg:765.27ms tok_s:10474
+step:3400/5000 train_loss:3.4664 train_time:2604591ms step_avg:766.06ms tok_s:10542
+step:3600/5000 train_loss:3.0442 train_time:2760180ms step_avg:766.72ms tok_s:10561
+step:3800/5000 train_loss:3.3471 train_time:2915808ms step_avg:767.32ms tok_s:10562
+step:4000/5000 train_loss:3.2545 train_time:3071637ms step_avg:767.91ms tok_s:10535
+step:4200/5000 train_loss:3.2583 train_time:3227249ms step_avg:768.39ms tok_s:10538
+step:4400/5000 train_loss:2.7846 train_time:3383286ms step_avg:768.93ms tok_s:10523
+step:4600/5000 train_loss:3.0846 train_time:3539220ms step_avg:769.40ms tok_s:10529
+step:4800/5000 train_loss:3.1119 train_time:3696477ms step_avg:770.10ms tok_s:10515
+step:5000/5000 train_loss:3.1048 train_time:3852578ms step_avg:770.52ms tok_s:10543
+step:5000/5000 val_loss:3.0670 val_bpb:1.5096 train_time:3852579ms step_avg:770.52ms
+    muon_momentum_warmup_steps: int = int(os.environ.get("MUON_MOMENTUM_WARMUP_STEPS", 500))
+        if self.args.muon_momentum_warmup_steps:
+            t = min(step / self.args.muon_momentum_warmup_steps, 1.0)
+    # The shard directory and tokenizer are coupled: val_bpb is only meaningful if we
+    # - val_bpb: tokenizer-agnostic compression metric used by the challenge
+    val_bpb = bits_per_token * (total_tokens / total_bytes)
+    return val_loss, val_bpb
+        f"model_params:{n_params} vocab_size:{args.vocab_size} layers:{args.num_layers} "
+        f"iterations:{args.iterations} train_batch_tokens:{args.train_batch_tokens} grad_accum_steps:{args.grad_accum_steps} "
+        f"warmup_steps:{args.warmup_steps} max_wallclock_seconds:{args.max_wallclock_seconds:.3f}"
+    log(f"val_bpb:enabled tokenizer_kind=sentencepiece tokenizer_path={args.tokenizer_path}")
+    if args.warmup_steps > 0:
+        for warmup_step in range(args.warmup_steps):
+            if args.warmup_steps <= 20 or (warmup_step + 1) % 10 == 0 or warmup_step + 1 == args.warmup_steps:
+                log(f"warmup_step:{warmup_step + 1}/{args.warmup_steps}")
+            val_loss, val_bpb = eval_val(
+                    f"step:{step}/{args.iterations} val_loss:{val_loss:.4f} val_bpb:{val_bpb:.4f} "
+            if stop_after_step is not None and step < args.iterations:
+    q_val_loss, q_val_bpb = eval_val(
+    log(f"final_int8_zlib_roundtrip val_loss:{q_val_loss:.4f} val_bpb:{q_val_bpb:.4f} eval_time:{q_eval_ms:.0f}ms")
+    log(f"final_int8_zlib_roundtrip_exact val_loss:{q_val_loss:.8f} val_bpb:{q_val_bpb:.8f}")
+model_params:33846448 vocab_size:4096 layers:11 dim:512 heads:8 kv_heads:4 seq_len:1024 tie_embeddings:True
+iterations:5000 train_batch_tokens:8192 grad_accum_steps:2 microbatch_tokens:4096 microbatch_batch_size:4 val_batch_size:524288 warmup_steps:10 max_wallclock_seconds:0.000
+val_bpb:enabled tokenizer_kind=sentencepiece tokenizer_path=./data/tokenizers/fineweb_4096_bpe.model
+warmup_step:1/10
+warmup_step:2/10
+warmup_step:3/10
+warmup_step:4/10
+warmup_step:5/10
+warmup_step:6/10
+warmup_step:7/10
+warmup_step:8/10
+warmup_step:9/10
+warmup_step:10/10
+step:10/5000 train_loss:7.0435 train_time:8189ms step_avg:818.90ms tok_s:10544
+step:200/5000 train_loss:4.5148 train_time:153865ms step_avg:769.33ms tok_s:10777
+step:400/5000 train_loss:4.3341 train_time:306493ms step_avg:766.23ms tok_s:10707
+step:600/5000 train_loss:4.3689 train_time:459261ms step_avg:765.43ms tok_s:10775
+step:800/5000 train_loss:3.9403 train_time:612662ms step_avg:765.83ms tok_s:10764
+step:1000/5000 train_loss:3.7845 train_time:765035ms step_avg:765.04ms tok_s:10750
+step:1200/5000 train_loss:3.7428 train_time:917720ms step_avg:764.77ms tok_s:10763
+step:1400/5000 train_loss:3.4893 train_time:1070595ms step_avg:764.71ms tok_s:10602
+step:1600/5000 train_loss:3.7819 train_time:1223754ms step_avg:764.85ms tok_s:10798
+step:1800/5000 train_loss:3.5227 train_time:1376010ms step_avg:764.45ms tok_s:10763
+step:2000/5000 train_loss:3.4185 train_time:1528915ms step_avg:764.46ms tok_s:10786
+step:2200/5000 train_loss:3.5589 train_time:1683208ms step_avg:765.09ms tok_s:10774
+step:2400/5000 train_loss:3.2930 train_time:1835796ms step_avg:764.91ms tok_s:10763
+step:2600/5000 train_loss:3.5559 train_time:1988350ms step_avg:764.75ms tok_s:10779
+step:2800/5000 train_loss:3.4060 train_time:2140679ms step_avg:764.53ms tok_s:10779
+step:3000/5000 train_loss:3.1498 train_time:2293002ms step_avg:764.33ms tok_s:10796
+step:3200/5000 train_loss:3.3820 train_time:2448866ms step_avg:765.27ms tok_s:10474
+step:3400/5000 train_loss:3.4664 train_time:2604591ms step_avg:766.06ms tok_s:10542
+step:3600/5000 train_loss:3.0442 train_time:2760180ms step_avg:766.72ms tok_s:10561
+step:3800/5000 train_loss:3.3471 train_time:2915808ms step_avg:767.32ms tok_s:10562
+step:4000/5000 train_loss:3.2545 train_time:3071637ms step_avg:767.91ms tok_s:10535
+step:4200/5000 train_loss:3.2583 train_time:3227249ms step_avg:768.39ms tok_s:10538
+step:4400/5000 train_loss:2.7846 train_time:3383286ms step_avg:768.93ms tok_s:10523
+step:4600/5000 train_loss:3.0846 train_time:3539220ms step_avg:769.40ms tok_s:10529
+step:4800/5000 train_loss:3.1119 train_time:3696477ms step_avg:770.10ms tok_s:10515
+step:5000/5000 train_loss:3.1048 train_time:3852578ms step_avg:770.52ms tok_s:10543
+step:5000/5000 val_loss:3.0670 val_bpb:1.5096 train_time:3852579ms step_avg:770.52ms
diff --git a/records/track_non_record_16mb/2026-04-14_MLX_Tuned_Hparams_1.5096/run_experiment.sh b/records/track_non_record_16mb/2026-04-14_MLX_Tuned_Hparams_1.5096/run_experiment.sh
@@ -0,0 +1,44 @@
+#!/bin/bash
+# EXP-048: SP4096 + Full SOTA + TUNED HYPERPARAMETERS
+# Applies Gemini+Codex joint recommendations:
+#  - MUON_MOMENTUM 0.99 → 0.95 (less aggressive, both AIs agree)
+#  - QK_GAIN_INIT 5.25 → 4.0 (Codex: too aggressive with partial RoPE)
+#  - MATRIX_LR 0.022 → 0.02 (Gemini: high leverage)
+# Baseline: EXP-042 val_bpb=1.5596
+cd /Users/lucaslt/Documents/side-gig/openai/parameter_golf/repo
+source .venv/bin/activate
+export RUN_ID=exp048_tuned_hparams
+export DATA_PATH=./data/datasets/fineweb10B_sp4096
+export TOKENIZER_PATH=./data/tokenizers/fineweb_4096_bpe.model
+export VOCAB_SIZE=4096
+export NUM_LAYERS=11
+export MLP_MULT=4
+export MODEL_DIM=512
+export NUM_HEADS=8
+export NUM_KV_HEADS=4
+export ITERATIONS=5000
+export WARMDOWN_ITERS=1000
+export WARMDOWN_KIND=cosine
+# TUNED values (was 0.022, 0.99, 5.25)
+export MATRIX_LR=0.02
+export TIED_EMBED_LR=0.03
+export SCALAR_LR=0.02
+export MUON_MOMENTUM=0.95
+export MUON_MOMENTUM_WARMUP_START=0.90
+export MUON_MOMENTUM_WARMUP_STEPS=60
+export GRAD_CLIP_NORM=0.3
+export QK_GAIN_INIT=4.0
+export TRAIN_BATCH_TOKENS=8192
+export VAL_LOSS_EVERY=0
+export VAL_BATCH_SIZE=524288
+export TRAIN_LOG_EVERY=200
+export MLX_MAX_MICROBATCH_TOKENS=4096
+export GRAD_ACCUM_STEPS=2
+export WARMUP_STEPS=10
+export MAX_WALLCLOCK_SECONDS=0
+# SOTA features (same as EXP-042 baseline)
+export RECUR_LAYERS=3,4,5
+export RECUR_START_STEP=0
+export PARALLEL_RESIDUAL=1
+export PARALLEL_START_LAYER=7
+python3 train_gpt_mlx.py