Record: SP8192 + Muon 0.97 + Legal Score-First TTT — val_bpb 1.07983 (3-seed mean)

[dexhunter]

Summary

• val_bpb: 1.07983 (3-seed mean, std 0.00050) / 2.78932 nats/token
• Artifact: ~15.99 MB (under 16 MB on all 3 seeds)
• Delta vs current merged SOTA #1493 (1.0810): 0.00117 bpb / 0.00302 nats/token

Builds on @clarkkev's PR #1394 sp8192 stack and our own PR #1413 legal score-first TTT, adding:

1. Muon momentum = 0.97 (vs 0.99 default) — single-knob hyperparameter sweep
2. Causal token n-gram tilt — prefix-only token expert from @abaybektursun's PR #1420 kernel (base_beta=2.0, agree_bonus=0.1); within-word and word-start experts explicitly disabled (within_beta=0, word_beta=0) because they cannot be made fully causal without losing most of the benefit.
3. Legal score-first TTT — already present in our PR Record: SP8192 + QK-Gain 5 + Legal Score-First TTT — val_bpb 1.08279 (3-seed mean) #1413

Results (8×H100 80GB SXM, PyTorch 2.9.1+cu128)

Seed	Pre-TTT sliding	Post-TTT bpb	val_loss (nats)	Artifact
0	1.08102	1.07928	2.78790	15,993,346
42	1.08167	1.07997	2.78967	15,992,995
1234	1.08194	1.08025	2.79039	15,994,604
mean	1.08154	1.07983	2.78932	15,993,648

std_bpb = 0.00050, std_nats = 0.00128. All 3 seeds fit the 16 MB artifact cap and complete under 600s train + 600s eval.

Legality

• Score-first TTT only — every sliding-window chunk is scored under inference_mode() before any gradient update. No chunk is trained on before scoring.
• Causal n-gram tilt — only the prefix-only token expert is active. The within-word and word-start experts from PR Record: Triple Loop + Fused Kernels + Parallel Residuals + N-gram Tilt; val_bpb 1.08309 (5-seed mean) #1420 are explicitly zeroed out. The kernel causality fix per the PR Record: Triple Loop + Fused Kernels + Parallel Residuals + N-gram Tilt; val_bpb 1.08309 (5-seed mean) #1420 author thread is applied.
• No SLOT, no pre-quant TTT on val data, no n-gram cache, no ETLB.
• Full 50k-doc val split, canonical ordering, single left-to-right pass.

Test plan

• 3-seed verification (seeds 0/42/1234)
• Artifact under 16 MB on all seeds
• Train under 600s on all seeds (~588s)
• Eval under 600s on all seeds (<437s)
• No val-data leakage in training
• Score-first TTT ordering verified
• Causal n-gram tilt verified (prefix-only metadata)