Non-record: 11L Depth Recurrence + High-Yield Legal TTT (1.14458 BPB)

[Christopher-Lee-McClendon]

Non-Record Submission: 11L Depth Recurrence + Legal Score-First TTT

val_bpb = 1.14458 | Pre-TTT: 1.1611 | TTT gain: −0.0165 | Artifact: 14.79 MB

Non-record unlimited-compute submission (trained on 4×A100-40GB).

---

Headline

This submission demonstrates that competition-legal test-time training can deliver large gains when properly tuned. The key finding is a TTT recipe — SGD with momentum, multiple epochs per chunk, freezing early blocks — that extracts 2.4× more improvement (−0.0165 BPB) than single-epoch AdamW over the full network (−0.0068 in our prior PR #456).

Every validation token is scored before any weight update that could use it, enforced by torch.inference_mode() during scoring.

Novel & Creative Contributions

1. High-yield legal TTT via selective freezing + SGD momentum — Most TTT approaches use AdamW for 1 epoch over all parameters. We use SGD+momentum(0.9) for 3 epochs per 32K chunk while freezing the first 2 blocks. This is simpler, uses less memory, and gets 2.4× better TTT gains.

2. Depth recurrence — 11 logical layers from 10 unique BlockCores (one core reused at two depths with independent normalization). Delivers 11-layer capacity at 10-layer parameter cost.

3. Partial RoPE (16/64 dims) — Only 16 of 64 head dimensions use rotary embeddings, with NTK-aware scaling. The remaining 48 act as position-agnostic content channels, improving length generalization during TTT.

4. Value Embeddings on deep layers only — 128-dim learned embeddings added to value projections on layers 9–10, giving deep layers direct token-identity access in the value stream.

5. Layer-Norm depth scaling — 1/√(layer+1) scaling stabilizes training under depth recurrence.

Architecture

Component	Setting
Layers	11 logical (10 shared BlockCores)
Dim/Heads	512 / 8 (4 KV heads)
MLP	3× (1536), ReLU²; SmearGate
BigramHash	2048
RoPE	16/64 dims, NTK scaling
Value Embed	128d on layers 9–10
XSA	Last 4 layers
Quant	Int6 + zstd

TTT Protocol

for each 32K chunk:
    1. model.eval() + inference_mode → score chunk (NLL recorded)
    2. model.train() → SGD(lr=0.002, mom=0.9), 3 epochs, freeze blocks 0-1

Size Budget

Component	Bytes
Model (int6+zstd)	14,717,713
Code	71,706
Total	14,789,419 (< 16,000,000)

vs. Prior Submission (PR #456)

	PR #456	This PR	Δ
val_bpb	1.15321	1.14458	−0.00863
Pre-TTT	1.1600	1.1611	+0.0011
TTT gain	−0.0068	−0.0165	2.4×

Pre-TTT baselines are nearly identical — the entire improvement comes from better TTT, validating the recipe.

Credits

This submission builds on work from many contributors to the parameter-golf competition:

• Muon optimizer — Baseline (modded-nanogpt); Newton-Schulz orthogonal preconditioning
• BigramHash embeddings — PR Record: Mixed Quant Int6/FP16 + SmearGate + OrthoInit + MLP 3x + Sliding Window, val_bpb=1.1556 #65 (aquariouseworkman)
• SmearGate — PR Record: Mixed Quant Int6/FP16 + SmearGate + OrthoInit + MLP 3x + Sliding Window, val_bpb=1.1556 #65 (aquariouseworkman)
• XSA — PR Record: EMA-GPU + Multi-Order N-gram Backoff + PE Confidence (val_bpb=0.9393) #187 (Idan3011); GQA-aware variant in PR Record: 11L + Efficient Partial XSA (val_bpb: 1.1307)  #265 (unnir)
• U-Net skip connections — PR Record: Mixed Quant Int6/FP16 + SmearGate + OrthoInit + MLP 3x + Sliding Window, val_bpb=1.1556 #65 (aquariouseworkman), PR SubSixteen v2: Int6 QAT + MLP 3x + SWA + Sliding Window (val_bpb 1.1708) #69 (TevBenji)
• Mixed int5/int6 quantization — PR 12L Int5-MLP + SmearGate + BigramHash + SWA (val_bpb 1.1433) #76 (unixmadtoonslab / Will DePue)
• SWA — PR SubSixteen v2: Int6 QAT + MLP 3x + SWA + Sliding Window (val_bpb 1.1708) #69 (TevBenji)
• Late QAT — PR Record: 11L Partial RoPE + LN Scale + EMA + XSA4 (val_bpb: 1.1248) #315 (jfprincz), PR Record: 11L + Tight SWA + Shared VE128 + Partial RoPE + LN Scale + XSA4 (val_bpb: 1.1246) #374 (unnir)
• Sliding window evaluation — PR Record: Sliding Window Eval (stride=64), val_bpb=1.1925 #50 (mattqlf / Matthew Li)
• Value Embeddings — PR Record: 11L + Tight SWA + Shared VE128 + Partial RoPE + LN Scale + XSA4 (val_bpb: 1.1246) #374 (unnir)
• Partial RoPE — PR Record: 11L Partial RoPE + LN Scale + EMA + XSA4 (val_bpb: 1.1248) #315 (jfprincz), PR Record: 11L + Tight SWA + Shared VE128 + Partial RoPE + LN Scale + XSA4 (val_bpb: 1.1246) #374 (unnir)
• LN Scale — PR Record: 11L Partial RoPE + LN Scale + EMA + XSA4 (val_bpb: 1.1248) #315 (jfprincz), PR Record: 11L + Tight SWA + Shared VE128 + Partial RoPE + LN Scale + XSA4 (val_bpb: 1.1246) #374 (unnir)
• Depth recurrence — PR Add depth recurrence + SwiGLU submission (Apple M3 8GB) #8 (iranzithierry)
• Legal TTT framework — PR [record bpb=1.195] sliding window + LoRA TTT #77 (samacqua); full-model variant in our PR Non-record submission: Depth Recurrence + Legal Score-First TTT (10L, 1.1532 BPB) #456
• ReLU², GQA — Baseline (modded-nanogpt)

Built on the parameter-golf starter code by Beren Millidge & Keller Jordan.

[mohosy]

the legal ttt protocol here is really well thought out, scoring before updating is the right way to do it. depth recurrence at this scale is underexplored too imo