11L SmearGate + BigramHash(10240) + Causal TTT + Mixed Int5/Int6 + SWA

[romainsantoli-web]

11L SmearGate + BigramHash(10240) + Causal TTT + Mixed Int5/Int6 + SWA

val_bpb: pending - awaiting RunPod credits for 8xH100 validation runs.

Approach

Combines the strongest proven techniques from top submissions into a unified architecture targeting sub-1.135 val_bpb:

• 11-layer GPT with U-Net skip connections, GQA (8h/4kv), RoPE
• SmearGate (PR Record: Int6 MLP3x + SmearGate + BigramHash + MuonWD + SWA (mean val_bpb=1.1483) #162): learned sigmoid gate for local context
• BigramHash(10240) (PR Record: Int6 MLP3x + SmearGate + BigramHash + MuonWD + SWA (mean val_bpb=1.1483) #162, Record: 10L Int5-MLP + BigramHash(10240) + SWA(0.4) + WD=0.04 (val_bpb=1.1428, mean 3 seeds) #180): explicit bigram statistics
• Causal TTT (PR Record: val_bpb: 1.14020 [tested 3x on 8xh100] #267, Non-record: val_bpb=1.1374, FA2+SWA adaptation of Farnsworth #281): test-time adaptation, each token scored before model sees it
• Mixed int5/int6 quantization (PR Record: 10L Int5-MLP + BigramHash(10240) + SWA(0.4) + WD=0.04 (val_bpb=1.1428, mean 3 seeds) #180): int5 for MLP, int6 for attn
• 3% magnitude pruning + zlib-9 compression
• SWA (frac=0.4, every=50 steps)
• Z-loss regularization (1e-4)
• Tuned HPs: WD=0.042, matrix_lr=0.025, grad_clip=0.3, warmdown=3000

Status

Code complete and syntax-validated. Requesting compute credits to run 3-seed validation on 8xH100.

Expected Results

Based on ablation data from PRs #162, #180, #267, #281 - target ~1.133-1.137 val_bpb.

Built on PR #162 by @unnir and PR #180 by @thwu1.