Record: MuonEq-R + Depth Recurrence + Mixed Int5/Int6 GPTQ — val_bpb 1.0929 (3-seed mean)

[dexhunter]

Summary

• val_bpb = 1.0929 (3-seed mean, std 0.0009) | 2.5145 nats | ~15.96 MB | 8xH100 SXM, 600s | No TTT
• Adds MuonEq-R optimizer, depth recurrence (layers 4,5 shared MLP), and mixed int5/int6 GPTQ to PR Record: 4096-Vocab + 4.0-MLP-mult + 0.085-WD + Simplifications — val_bpb 1.09785 (3-seed mean) #1218's 4096-vocab high-WD stack
• All 3 seeds under 16MB (max: 15,981,324 bytes)
• No SLOT, no eval-time adaptation, fully legal

Key Innovations

1. MuonEq-R — Row-normalizes gradient matrices before Newton-Schulz orthogonalization. Zero-byte cost, ~0.001 BPB improvement.
2. Depth Recurrence — Layers 4,5 repeated with fully shared MLP weights (zero extra params). ~0.003 BPP improvement.
3. Mixed Int5/Int6 GPTQ — Hessian sensitivity ranking: 60 int6 + 6 int5 layers for optimal size/quality tradeoff.

Results (8xH100 80GB SXM, PyTorch 2.9.1+cu128)

Seed	Steps	ms/step	Sliding BPB	val_loss (nats)	Artifact
1337	5,541	106.5	1.0939	2.51667	15,933,457
42	5,530	106.7	1.0922	2.51279	15,981,324
0	5,543	106.5	1.0927	2.51394	15,960,050
Mean	5,538	106.6	1.0929	2.51447	15,958,277

Changes from PR #1218

	PR #1218	This
val_bpb	1.09785	1.09290 (-0.00495)
val_loss	~2.526 nats	2.514 nats (-0.011)
Optimizer	Muon	MuonEq-R
Depth recurrence	None	Layers 4,5
Mixed quantization	No	60 int6 + 6 int5

Credits

• @clarkkev for PR Record: 4096-Vocab + 4.0-MLP-mult + 0.085-WD + Simplifications — val_bpb 1.09785 (3-seed mean) #1218 (4096-Vocab + MLP 4x + WD 0.085 — the foundation)
• @abaybektursun for PR Record: AR Self-Gen GPTQ + XSA-all + BigramHash 3072×112 — val_bpb 1.11473 (3-seed mean) #1019 (GPTQ + XSA + BigramHash baseline)
• @msisovic for PR Record: ParallelResiduals + MiniDepthRecurrence, 1.1063 BPB / 1.8679 nats, -0.0072 vs PR #1179, -0.0143 vs merged SOTA #1204 (depth recurrence concept)

Test plan

• 3-seed verification (1337, 42, 0) — all pass artifact + time + score
• All seeds under 16,000,000 bytes
• Train < 600s, eval < 600s
• No TTT, no SLOT, no forbidden techniques
• Rule checker passed (log + script)

[mikeapedia]

Great submission @dexhunter! Did you happen to test muon column norm or row+column norm? I found R+C worked the best with the smaller vocab and I am wondering if that holds here as well.