Skip to content

Non-Record: 11L Parallel Muon + LN Scale + LeakyReLU² MLP3x + Legal TTT — val_bpb 1.1215 (3-seed mean)#838

Open
aryanbhosale wants to merge 1 commit intoopenai:mainfrom
aryanbhosale:submission/11l-parallel-muon-lnscale-ttt
Open

Non-Record: 11L Parallel Muon + LN Scale + LeakyReLU² MLP3x + Legal TTT — val_bpb 1.1215 (3-seed mean)#838
aryanbhosale wants to merge 1 commit intoopenai:mainfrom
aryanbhosale:submission/11l-parallel-muon-lnscale-ttt

Conversation

@aryanbhosale
Copy link
Copy Markdown
Contributor

Non-Record: 11L Parallel Muon + LN Scale + LeakyReLU² MLP3x + Legal Score-First TTT

val_bpb = 1.1215 (3-seed mean, std 0.0002) | ~15.85 MB | 8×H100 SXM

3-Seed Results (8×H100 80GB SXM, PyTorch 2.9.1+cu128)

Seed step_avg steps EMA bpb Quantized bpb TTT bpb
1337 88.8ms 6,759 1.1161 1.1238 1.1217
42 88.8ms 6,757 1.1158 1.1234 1.1213
2024 88.9ms 6,752 1.1160 1.1234 1.1215
Mean 88.8ms 6,756 1.1160 1.1235 1.1215

Key Techniques

  1. Parallel Muon with parameter banking — 4 contiguous 3D banks, batched Newton-Schulz via torch.bmm, 3-phase async reduce-scatter/all-gather. ~89ms/step, ~6750 steps.
  2. LN Scale — depth-dependent normalization: 1/sqrt(layer_idx+1). Stabilizes deep layers, -0.003 BPB.
  3. LeakyReLU(0.5)² MLP 3x — preserves negative gradient flow.
  4. Legal Score-First TTT (PR Non-record: 11L Depth Recurrence + High-Yield Legal TTT (1.14458 BPB) #461/Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean) #549 recipe) — score each 32K-token chunk with sliding windows under inference_mode, then SGD(lr=0.002, momentum=0.9) for 3 epochs, all blocks unfrozen, cosine LR decay.
  5. EMA(0.997) + SWA — EMA selected as best pre-quant weights (1.1160 BPB mean).
  6. GPTQ-lite int6 + zstd-22 — per-row 5-percentile clip search, FP16 embedding passthrough.
  7. Flash Attention 3, torch.compile(fullgraph=True), no DDP.

Architecture (26.8M params)

11L, 512d, 8H/4KV (GQA), MLP 3x, LN Scale, SmearGate, BigramHash(1536), Value Residual, Gated Attention, XSA4, Partial RoPE(16/64), U-Net skips, OrthoInit, tied embeddings, logit softcap 30.0.

Training

  • Parallel Muon: lr=0.025, momentum 0.92->0.99/1500 steps, WD=0.04, Newton-Schulz 5 steps
  • Adam for embeddings (lr=0.035) and scalars (lr=0.025)
  • Batch 786,432 tokens, seq_len 2048, warmdown 3500 iters
  • Late QAT via STE (final 15% wallclock)
  • Gradient clipping 0.3

TTT Compliance

Legal score-first per PR #461/#549 framework:

  • Every token scored BEFORE any weight update (enforced by torch.inference_mode)
  • No training data access during evaluation
  • No multi-epoch scoring -- each chunk scored exactly once
  • Total eval time: ~530s (within 10 min limit)

Credits

@aryanbhosale aryanbhosale force-pushed the submission/11l-parallel-muon-lnscale-ttt branch from 5687f3c to 26045f3 Compare March 27, 2026 06:31
aryanbhosale added a commit to aryanbhosale/parameter-golf that referenced this pull request Mar 28, 2026
…ope 0.75, no SWA, QAT50%, mHC, LZMA, SGD TTT
@MatoTeziTanka
Copy link
Copy Markdown

Community Review — Non-Record: 11L Parallel Muon + LN Scale + LeakyReLU² MLP3x + Legal TTT — val_bpb 1.1215 (3-seed mean)

BPB: 1.1215 | Compliance: LOOKS CLEAN — score-first-per-chunk TTT (legal #1416/#1423 pattern)

What I found in the code (head SHA 26045f3ac998, file records/track_10min_16mb/2026-03-25_11L_ParallelMuon_MLP3x_TTT/train_gpt.py):

The TTT path at line 423 implements the score-first-per-chunk pattern: each chunk is scored under torch.no_grad() / inference_mode() before the base_model.train() + SGD adaptation runs on that same chunk, with an is_last_chunk guard so the final chunk gets no adaptation pass. This is the structural shape the legal frontier uses (PRs #1416 erichroepke, #1423 aryanbhosale).

Per Issue #402 and Issue #677, TTT is legal when each token is scored before the adapter updates on it, and that's what the code does here — chunk ci is scored under weights adapted only on chunks 0..ci-1. No prequant_ttt_adapt_adamw(val_tokens, ...) multi-epoch fine-tune, no scored-region SLOT, no target-in-key n-gram cache.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.03s, dim=512, layers=11, vocab=1024, code=81870 B, SMOKE_TEST_PASS

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending standard checks (3-seed validation, 16MB artifact cap, 10-min wallclock on 8×H100 SXM). The compliance picture matches the legal reference frontier and no flags were raised by the classification pass.

Auto-classification caveat: this review was drafted by the AST-based classifier against a template derived from manually-reviewed cluster PRs (#1420, #1450, #1487, #1541, #1529, #1533, #1518). If I've misread a subtlety in your eval path — e.g., multi-epoch TTT that I mistook for single-pass, or a target-in-key lookup I missed in a helper function — please flag it and I'll re-run the audit manually.


Reviewed by @MatoTeziTankaThe Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.03s, dim=512, layers=11, vocab=1024, code=81870 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants