You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Parameter Golf Live AI Commentary (Inactive as of Apr. 1st)
Auto-updated every ~10 minutes. Tracking techniques, trends, idea lineage, and explaining concepts for the community.
Last updated: Apr 5, 7:56 PM PT
The Competition at a Glance
Goal: Train the best language model that fits in a 16MB artifact, training in under 10 minutes on 8xH100s. Evaluated by compression of the FineWeb validation set, measured in bits per byte (BPB) — lower is better. Tokenizer-agnostic. Baseline: 1.2244 BPB.
What does "compression" mean here?
BPB (bits per byte) measures how many bits your model needs to encode each byte of text. A model that perfectly predicts every next character needs zero bits — it already "knows" what comes next. A model with no understanding of language needs the maximum (~8 bits per byte).
A model's cross-entropy loss IS its compression rate. Shannon proved in 1948 that prediction and compression are mathematically equivalent — a model that predicts well compresses well, and vice versa. The competition measures the compression side of that equivalence.
This framing matters because it legitimizes approaches beyond pure language modeling: sliding window eval improves compression by giving more context. Backward-looking TTT adapts to already-scored tokens for better compression. These are valid compression strategies.
There is no separate held-out test set — the FineWeb validation set is the fixed evaluation target. However, val tokens cannot be stored in the artifact (paid prefix ruled out), and pre-eval adaptation on val data is also ruled out. Only backward-looking TTT (adapting on tokens already graded) is permitted.
"Tokenizer-agnostic" means BPB normalizes across tokenizers. A bigger vocabulary uses fewer tokens but more bits per token — BPB cancels that out, measuring compression of raw bytes regardless of how they're tokenized.
Record submission requirements: Artifact ≤16,000,000 bytes (code + compressed model). Training ≤10 min on 8xH100 SXM. Evaluation ≤10 min (separate budget). No network calls. New SOTA records must beat the current best by ≥0.005 nats at p < 0.01 significance (typically 3 seeds). Evaluation methods are unrestricted — any sequence length, sliding window, etc. are fair game. Test-time training is allowed only on already-evaluated tokens (backward-looking); pre-eval adaptation on val data is ruled out.
The competition launched Mar 18. The official SOTA is now 1.1147 BPB (#1019, @abaybektursun, Mar 25 — merged, AR Self-Gen GPTQ + XSA-all). The best pending submission overall is #1229 (@resouer, 0.9300 BPB, 3-seed) using Scored-Position SLOT + Per-Sample Delta + training-data GPTQ. Previously #1184 (@icryo, 0.9485) using Scylla tokenizer (998-vocab TokenMonster) + Full GPTQ + XSA-all. The best standard-tokenizer submission is #1176 (@bigbag, 1.0914, 3-seed) with QK-Gain 4.0 + Muon-TTT + SLOT. A new n-gram backoff cache submission #1185 (@skoustav35, 0.9641) claims proper Laplace normalization. @abaybektursun's #1105 (1.1125, CUTLASS EVT + MLP 3.5x + mixed int5/int6) narrowly misses the 0.005-nat record threshold vs the new SOTA. @dentity007 submitted 7 PRs addressing README wishlist items (JEPA, text diffusion, H-Net, Universal Transformer, SSM, megakernels, random linear maps) — all proof-of-concept stage. The SOTA shift from 1.1194→1.1147 means 4 former record submissions (#728, #1060, #1099, #1130) no longer meet the 0.005-nat threshold.
Blue = official leaderboard (1.2244 → 1.1147). Green = pending frontier (now invalidated n-gram submissions removed). Red dashed = official SOTA (1.1147, #1019).
New official SOTA: 1.1147 (#1019, merged Mar 25). Best pending: #1184 (0.9485, Scylla tokenizer), #1185 (0.9641, n-gram backoff cache), #753 (0.9625, Podracing II). Best standard tokenizer: #1176 (1.0914, QK-Gain + TTT + SLOT). Four former records (#728, #1060, #1099, #1130) downgraded — no longer clear the 0.005-nat threshold against the new SOTA. Tables below ↓
Pending: Meets Record Requirements
Record-eligible submissions only. Pre-eval TTT entries excluded per @0hq ruling on #402 — only backward-looking (score-first, single-pass) TTT is allowed. Official SOTA: 1.1147 BPB (#1019, @abaybektursun — AR Self-Gen GPTQ + XSA-all + Legal TTT + Parallel Muon, updated Mar 24).
Top 5 record-eligible (30 total — full table in collapsible below):
Complementary Training — tokens predictable by bigram stats get lower loss weight during training. Model specializes on what n-grams can't predict, enabling higher eval-time n-gram alpha (20-75%). + Backoff N-gram Mixer + VRL + XSA-4. Std=0.0001.
Seed-Regenerated Random Model + Incremental N-gram Cache. Model weights generated from seed (not trained) — neural baseline 1.503 BPP. All compression from n-gram cache. 1-seed only, run on MI250X (not H100). Pending H100 validation + 2 more seeds.
30 record-eligible + 100 unvalidated | Official SOTA: 1.1147 (#1019) | Full tables in collapsibles below ↓
Note: The full "All Pending Validated" table below contains the pre-n-gram-cache entries. N-gram/Hedge Mixer submissions still open (#702, #715, #706, #700) and #728 is tracked in the Not Yet Validated table (downgraded when SOTA moved to 1.1147).
Untried Combinations
Ranked by expected value (likely gain times probability of working), grounded in competition ablation data:
Tier 1 — Highest expected value (n-gram cache extensions)
Context Tree Weighting (CTW) instead of heuristic alpha. The current top n-gram submissions use hand-tuned or entropy-adaptive alpha to mix n-gram orders. CTW (Willems et al.) provides Bayesian-optimal weighting over all context tree models up to a given depth — provably minimax optimal for tree sources. Replaces heuristic with theory. Zero artifact cost. Non-Record: SLOT Eval-Time Augmentation on PR #549 SOTA Stack val_bpb = 1.1185 (3-seed mean, std 0.0003) | ~15.9 MB | 8×H100 SXM #1084 tested depth-4 CTW: +0.005 BPB worse, 46 min eval — negative result. Original estimate questionable. Moderate complexity (tree data structure).
Logistic-domain mixing. Current submissions use linear interpolation: alpha*p_ngram + (1-alpha)*p_neural. PAQ-style compressors use log-odds space mixing, which handles extreme probabilities better. A one-line change. Est. 0.002-0.005 BPB. Trivial complexity.
Adaptive stride (entropy-guided two-pass). First pass with stride=64 scores all tokens and records per-token entropy. Second pass re-evaluates high-entropy regions with smaller stride (16-32). Targets compute where it helps most. Backward-looking, zero artifact cost. Est. 0.005-0.015 BPB. Low-moderate complexity.
Fixed-Share Hedge (non-stationary expert tracking).Record Submission: 1.0541 BPB - 5-expert Hedge Mixer + CROWN-Q + stride=64 #700's Hedge algorithm assumes stationary expert quality. Fixed-Share (Herbster & Warmuth) allows "switching" between experts — important because FineWeb contains diverse content types (code, prose, tables). Zero artifact cost. Est. 0.003-0.008 BPB over standard Hedge. Low complexity (one parameter: switching rate).
PPMII-style escape estimation (Shkarin 2001). Replace heuristic backoff with principled escape probabilities + information inheritance (new context nodes inherit counts from parent) + full exclusions (symbols already assigned probability at order-k excluded from order-(k-1)). The theoretically optimal version of what the current BackoffNgramMixer approximates. 40 years of compression research refinement. Est. 0.01-0.03 BPB. Medium complexity.
Match model (longest-match prediction). Instead of fixed high-order n-grams (8, 9, 10+), find the longest match anywhere in previously-scored data and predict based on what followed that match. Used by LPAQ/PAQ. Captures arbitrarily long repeated contexts without exponential memory cost. Complements rather than replaces multi-order backoff. Est. 0.005-0.01 BPB. Medium complexity.
Sparse/skip-gram context models. Use non-contiguous positions (e.g., tokens at -1, -3, -5) as context. Captures patterns with intervening variable content (HTML tags, code indentation, sentence structures). Multiple sparse models with different gap patterns. Zero additional memory per context — just hash different positions. Especially effective on FineWeb's structured web text. Est. 0.005-0.015 BPB. Low complexity.
Cascaded SSE stages (3-5 chained APMs). Rather than a single SSE post-processing step, chain multiple Adaptive Probability Map stages with progressively higher-order contexts. Each stage corrects residual biases from the previous one. Used by PAQ/PAQAR. Est. 0.005-0.015 BPB. Low complexity.
Complementary distillation — add -lambda * KL(P_ngram || P_model) to the training loss. Pushes the neural model to explicitly diverge from the n-gram distribution at every token, not just binary loss reweighting. Based on "N-gram Is Back" (Li et al., EMNLP 2022) residual learning framework. Smooth, differentiable. Est. 0.01-0.03 BPB over current Complementary Training. Low complexity.
Three-tier token weighting — extend Complementary Training beyond binary easy/hard. Down-weight tokens predictable by bigrams (n-gram handles them), AND down-weight "noise" tokens (random proper nouns, typos) that neither model will ever predict well. Concentrate gradient on the learnable frontier. Based on Token Weighting for Long-Range LM (NAACL 2025). Est. 0.005-0.015 BPB. Low complexity.
Higher-order complementary training — Record: 0.4416 BPB -- Complementary Training + Backoff N-gram Mixer #803 uses bigram statistics for loss reweighting. Using 4-gram or 7-gram statistics (matching the actual eval-time cache) would better align training with eval. Tokens easy for 7-grams but hard for bigrams currently get full training weight but will be handled by the cache at eval time. Est. 0.005-0.010 BPB. Low complexity.
Adaptive complementary alpha — instead of fixed COMPLEMENT_ALPHA=0.5, make w = 1 - alpha * p_ngram. Tokens with p_ngram=0.99 get near-zero weight; tokens with p_ngram=0.1 get nearly full weight. Smooth weighting instead of hard threshold. Est. 0.003-0.008 BPB. Trivial complexity.
GuidedQuant (gradient-aware PTQ). Integrates end-loss gradients into layer-wise quantization objectives, improving over standard GPTQ. Could replace the current GPTQ calibration step with gradient-informed block selection — relevant because GPTQ calibration quality directly affects the int5/int6 roundtrip loss. No competition submissions have tried this yet. Expected gain: 0.002–0.005 BPP over standard GPTQ (based on paper claims of consistent PTQ improvement). Source: OpenReview 2025.
Frequency-ordered tokenization. Reorder vocabulary by frequency and encode with variable-length integers before compression. Achieves 0.76–7.08 percentage point improvements on standard compressors (zlib, LZMA, zstd). Directly applicable as a post-hoc encoding step on the artifact — no training changes needed. Could save 200–500KB of artifact budget, enabling a larger model. Source: arXiv 2602.22958.
Larger model at lower bits (2-bit). Recent research confirms: a larger model quantized to 2-bit outperforms a smaller model at 4-bit given a fixed compression budget. Current competition meta is 11L/512d at int5/int6. A 14L/640d model at int3 could fit in 16MB and outperform — but requires int3 quantization infrastructure nobody has built yet. High risk, high reward.
Tier 2 — Top picks for pure neural track (sorted by expected value)
Engram — TRIED as EngramLite (Record Submission: 1.1091 BPB - Turbo-Muon + EngramLite + ParamBanking + XSA (11L 512d) #1089: bigram+trigram, 2 heads, 8192 buckets, part of 1.1086 record) (DeepSeek, Jun 2025). Multi-head hashing (K=4 heads per N-gram order) + context-aware gating (sigmoid gate suppresses noisy lookups) + tokenizer compression (collapse equivalent IDs, −23% vocab). The competition's BigramHash is a primitive single-head version. Engram's gating could rescue higher-order N-grams (Non-record: 11L XSA-all + Full GPTQ + Selective Pruning (val_bpb=1.1154, 3-seed) #609 showed TrigramHash hurts without gating: +0.0049). Multi-head reduces hash collisions. Depthwise causal conv for temporal smoothing. Main constraint: embedding tables consume artifact space (2-4MB for full multi-order). Est. 0.003-0.008 BPB.
Mousse optimizer (arXiv:2603.09697). Curvature-aware Muon — Shampoo preconditioning before orthogonalization. ~12% more effective training at 3% overhead. Drop-in. Est. 0.003-0.008 BPB.
OptRot pre-quantization (arXiv:2512.24124). Rotation matrix redistributes weight outliers before quantizing. Fuses into adjacent layers — zero artifact cost. Est. 0.001-0.003 BPB (reduced estimate — Full GPTQ already handles much of the outlier problem; 11L + Hadamard Rotation + VE128 + cuDNN SDPA (val_bpb: 1.1365, 3-seed mean) #586 shows rotation "substitutes with GPTQ at int6").
qTTT — query-only test-time training (arXiv:2512.13898). Cache K/V once, adapt only Q projection weights. 2-3x more TTT epochs within eval budget. Est. 0.003-0.010 BPB. Note: use AdamW with cosine LR, not SGD — Non-record: VR + GA + Late QAT + Full GPTQ — 1.1418 BPB, 15.7 MB #601 shows SGD TTT hurts GPTQ models (+0.030). AdamW TTT works but requires GPTQ calibration within training budget (not eval time).
LaCT — Large Chunk TTT (arXiv:2505.23884, ICLR 2026 Oral). Document-sized chunks → 70% GPU utilization (vs <5% for per-token TTT). Uses Muon as fast-weight optimizer. Est. 0.002-0.008 BPB over current TTT approaches.
Prune-then-quantize ordering (arXiv:2603.18426, ICLR 2026). Progressive Intensity Hypothesis: weaker perturbations first, stronger later. Non-record: 11L XSA-all + Full GPTQ + Selective Pruning (val_bpb=1.1154, 3-seed) #609 currently does quantize-then-prune; reversing the order is a zero-cost experiment. Theory + experiments show 0.001-0.003 BPB free gain.
relu² is already 84-98% sparse; enforce NVIDIA 2:4 pattern for 2× sparse matmul on H100 tensor cores. ~15-20% more training steps. Systems-only = significance waived.
Moderate (custom kernels)
In-Place TTT with NTP objective (ICLR 2026 Oral)
0.003-0.010
Update MLP final projections during eval using NTP loss (not reconstruction). NTP alignment may explain why naive SGD TTT is neutral at frontier — objective misalignment. MLP-only, last 3 blocks.
Moderate
PoPE — Polar Position Embedding (arXiv:2509.10534)
0.002-0.005
Decouples content (magnitude) from position (angle) in attention. Principled fix for what Partial RoPE approximates. Strong length extrapolation. OpenAI co-author.
Adjacent layer pairs share K/V projections. Saves ~0.5MB artifact for 12L or wider MLP. Unlike depth recurrence, only K/V shared — no quant amplification.
Moderate
Block AttnRes (arXiv:2603.15031, Kimi, Mar 2026)
0.003-0.008
Efficient variant of AttnRes (which failed at 54% overhead in #362). Block partitioning (3 blocks at 11L) reduces overhead to <2%. 1.25× convergence efficiency.
Moderate
QK-Norm (arXiv:2010.04245, used in Gemma 2/DeepSeek-V3)
0.001-0.004
L2-normalize Q and K before dot product + learned per-head temperature. Prevents attention logit explosion — the root cause LN Scale patches. Could enable stable 12-13L training. Suppresses #215's Q condition numbers (100M+ → 1). ~4 lines.
Very low
Hourglass FFN (arXiv:2602.06471, Feb 2026)
0.002-0.006
Replace wide MLP-3x with stacked narrow-to-narrow sub-MLPs + residuals. Deeper MLP at fewer params. Paper: outperforms conventional FFN up to 400M params. Freed params → extra layers or larger BigramHash.
Low-moderate
CERWU (arXiv:2505.18758)
0.003-0.008
Rate-distortion optimal quantization: co-optimizes quant grid + weight updates + entropy coding. GPTQ is special case (λ=0). Principled upgrade to GPTQ-lite. Post-training, orthogonal to QAT.
Start with short local attention (128-384 tokens), grow to full 2048 during training. Faster early steps → more total steps. Different from blocked seq curriculum — same input length, just restricted attention span. Systems-only.
Moderate
NuMuon (arXiv:2603.03597, Mar 2026)
0.002-0.006
Nuclear-norm constraint on Muon updates → lower stable rank → better zstd compression. Pushes compressibility into optimizer itself. Distinct from Mousse/Turbo-Muon (those target speed).
Low-moderate
AdamHD Huber Decay (arXiv:2511.14721)
0.002-0.005
Replace L2 weight decay with Huber regularizer: quadratic below threshold, linear above. Specifically suppresses large outlier weights that cause int6 clipping loss. Drop-in for Muon's decoupled WD. Synergizes with GPTQ-lite (fewer outliers = less work).
Very low
Layer-Wise Scaling (arXiv:2509.06518)
0.002-0.005
Non-uniform FFN width per layer (e.g., MLP-4x middle, MLP-2x edges). Same total params, better allocation. Crown/Frame/Reverse variants all beat uniform at 180M params. Complements Hourglass FFN (structure vs width). Zero cost — just per-layer dims.
Learned multi-depth residual mixing: replaces x+f(x) with a connection matrix (n=2 → 16 params/layer, ~176 total). Richer than Catalytic Residuals or DenseFormer DWA. mHC adds Sinkhorn stability. Drop-in.
Low-moderate
HESTIA soft QAT (arXiv:2601.20745)
0.002-0.006
Replaces hard STE with temperature-annealed softmax relaxation + per-tensor Hessian guidance. Enables earlier QAT without premature discretization. Synergizes with OptRot.
Moderate
Compute-Optimal QAT (arXiv:2509.22935, Apple)
0.001-0.004
Scaling law for optimal FP→QAT split. Cooldown+QAT fusion: activate QAT at warmdown onset, eliminating redundant FP updates. Principled replacement for empirical Late QAT thresholds.
Very low
ScaleBITS (arXiv:2602.17698)
0.002-0.006
Automated per-layer bit-width search (which layers get int5 vs int6). Sensitivity analysis + greedy optimization under 16MB constraint. +36% over uniform precision in paper. Replaces manual assignment.
Moderate
CPSVD (arXiv:2510.19385)
0.003-0.008
Column-Preserving SVD: identify weight columns that compress cleanly via low-rank factorization, store rest as int6. Orthogonal to quantization — reduces param count, not precision. Freed bytes → capacity. Entirely unexplored in competition.
Moderate
Softpick / Rectified Softmax (arXiv:2504.20966)
0.002-0.006
Replaces softmax with rectified non-sum-to-one variant. Eliminates attention sinks and massive activations — directly improves int-N quantization quality (lower kurtosis). 47% sparse attention maps. "Quantized Softpick outperforms quantized softmax at lower bit widths."
Low
Anti-Layer Removal (arXiv:2603.19348)
0.002-0.006
Some layers are "anti-layers" whose removal improves performance. Anatomical analysis of 135M model shows 10^7 importance range. If 1-2 middle layers of 11L are anti-layers, removing them frees artifact space for wider MLP or more BigramHash. Zero-cost ablation pass on existing checkpoint.
Very low
Deep Delta Learning (DDL) (arXiv:2601.00417)
0.003-0.007
Rank-1 erasure gate on residual: x + β·proj(x) + f(x). Learned gate erases stale features before writing new ones. 3-5 ppl improvement at 124M. ~5.6K params for 11L. Addresses residual-path interference in quantized models.
Variance normalization before NS orthogonalization. Reduces Muon's step-size sensitivity + hyperparameter sensitivity. Zero extra hyperparameters — direct drop-in. Lower val loss than standard Muon on GPT-2/LLaMA.
Very low
TEON cross-layer Muon (arXiv:2601.23261)
0.003-0.007
Joint tensor orthogonalization across ALL layers (vs Muon's per-layer NS). Captures inter-layer gradient relationships. Consistent ppl improvement 130M-774M. Targets loss per step — critical for 600s budget.
Moderate
Seesaw LR+Batch Schedule (arXiv:2510.14717)
0.002-0.005
Multiply LR by 1/sqrt(2) and double batch size simultaneously. ~36% fewer serial steps at equal FLOPs. Principled foundation for the 524K→786K ramp.
Replace linear warmdown with 1-sqrt((t-T0)/(T+1-T0)). Outperforms linear, cosine, and other cooldown shapes in WSD schedules. Zero-cost swap.
Very low
SSMax (Scalable-Softmax) (arXiv:2501.19399)
0.001-0.004
Scale softmax by input sequence length to prevent attention flattening at seq2048. One scalar multiply. Compatible with FA3.
Very low
DCMHA (arXiv:2405.08553, ICML 2024 Oral)
0.005-0.015
Dynamically Composable Multi-Head Attention. Input-dependent transforms on score/weight matrices. Matches 1.7-2x compute models at 405M. Few KB params for 11L.
Moderate-high
VPTQ (arXiv:2409.17066, EMNLP 2024)
0.002-0.006
Vector PTQ guided by second-order Hessian. Beats GPTQ by 0.01-0.34 ppl at 2-3 bits. 10-18x faster than AQLM. Practical within 600s budget.
Trellis coded quantization — stateful sequential coding achieving ultra-high-dimensional VQ. At 3 bits, matches GPTQ at 4 bits. Bitshift trellis for GPU-parallel decoding.
High
Context Tree Switching (CTS)
0.002-0.008
Extension of CTW that handles non-stationary sources (distribution shifts between documents). Same complexity as CTW but mixes over larger model class.
Moderate
Tier 3 — Novel approaches, higher risk
Knowledge distillation — untried. Train larger teacher ~7 min, distill to 16MB student ~3 min. Est. 0.005-0.010 BPB but tight time budget is the constraint. High complexity.
Partial weight sharing + 14L — share middle-layer pairs with per-layer LoRA adapters. Saves 3-5 MB for extra layers. "Relaxed Recursive Transformers" shows LoRA-adapted shared layers recover most unique-layer quality. Est. +0.005-0.015 BPB. The Frugendorff: Recursive Weight Sharing Research — Cadence Laws + Challenges (1.1325 BPB) #579 tested 6×2 loops: 1.1478 but GPTQ compounds multiplicatively (see What Doesn't Work).
nGPT hypersphere normalization (arXiv:2410.01131) — constrain Q/K to unit-norm rows, eliminating Non-Record: 11L Low-Rank on Q192 (val_bpb=1.1548) 14.7MB in decimal #215's extreme condition numbers. NVIDIA claims 4-20x convergence speedup. Est. 0.003-0.008 BPB. High complexity, untested at this scale.
Drop-in Muon replacement using triangular whitening instead of Newton-Schulz. 1.3-2.6x faster peak tokens/s on A100.
0.001-0.003 BPB (via 200-500 extra steps)
Easy
Sigmoid attention + FlashSigmoid (ICLR 2025)
Replace softmax with element-wise sigmoid. FlashSigmoid kernel: 17% inference / 4% training speedup on H100. Eliminates token competition.
0.001-0.005 BPP (speed + possible quality gain)
Moderate
Entropy coding for weights (ANS/Huffman vs zstd)
Specialized entropy coders exploit quantized weight statistics better than general-purpose zstd. #1089 used Brotli+byte-shuffle instead of zstd on mixed int6/int7. EntQuant achieves 2-3 bit effective rates from FP8.
0.005-0.015 BPP (20-40% smaller artifacts = larger model or higher precision)
Moderate
Compute-Optimal QAT scheduling (Apple 2025)
FP cooldown + QAT fusion — skip separate cooldown phase, do LR decay jointly with QAT. Optimal QAT fraction depends on compute budget.
0.001-0.003 BPP
Easy
Medium relevance — higher implementation cost:
Technique
What It Is
Est. Impact
Difficulty
QTIP (NeurIPS 2024)
Trellis coded quantization — Viterbi-optimal paths through 256-dim codebook. Near-FP16 quality at 2-bit.
0.005-0.015 BPP (fit 3x more params in 16MB)
Hard
Mixture of Recursions (NeurIPS 2025)
Per-token adaptive depth in recursive transformers. Easy tokens exit early; hard tokens get more passes. 2x inference throughput.
0.002-0.008 BPP (more TTT iters or longer windows)
Brotli quality=11 outcompresses LZMA-9 by ~580KB on int6 weights. Byte-shuffle pre-filter groups MSB/LSB bytes for better compression. 580KB = ~93K extra int5 params.
Tokenizer Optimization (Validated by #1143 Scylla — 1.0806 BPP)
The biggest single-technique gain in the competition came from tokenizer choice, not architecture or quantization. #1143 achieved 1.0806 BPP (beating #1089's 1.1086 by 0.028) primarily through a TokenMonster-derived tokenizer.
Why it matters disproportionately at this scale: The embedding layer costs V*d parameters. At V=1024, d=384: ~295KB quantized. At V=32K: ~9.2MB — over half the 16MB budget. The parameter golf regime makes vocabulary size a first-order architectural decision.
TokenMonster vs BPE/Unigram: TokenMonster uses ungreedy multi-branch search (6 parallel branches scored per position). Produces ~37.5% fewer tokens at equivalent vocab size compared to BPE. BPE's greedy merges are known suboptimal (Bostrom & Durrett, EMNLP 2020).
Compression is necessary but not sufficient: Schmidt & Reddy (EMNLP 2024) showed maximum compression does not maximize model performance. The winning tokens are those aligned with what the model can learn, not those that minimize sequence length. #1143's autoresearch found a better vocabulary than pure compression optimization.
Open directions:
Direction
What to Try
Est. Impact
More aggressive vocab pruning
Literature suggests 60%+ tokens removable. #1143 only pruned 2.5% (1024→998). Try 800? 512?
0.005-0.02 BPB
FineWeb-aligned tokenizer training
Train tokenizer on FineWeb itself, not generic English. Reduces domain mismatch.
0.002-0.01 BPP
Byte-level fallback with larger vocab
TokenMonster + 2048 vocab. More tokens but each carries more information.
Turbo-Muon + EngramLite + Scylla. Currently untried.
Potentially <1.07 BPP
Techniques from Latest Competition Frontier (Mar 31)
Technique
What It Is
Where Used
Est. Impact
QK-Gain scaling (gain=4.0)
Learnable per-head scalar on queries after QK-norm. Controls attention sharpness (temperature). Higher gain = sharper attention = more decisive routing. Zero parameter cost.
#1176 (-0.004 BPP from #1125's 45-experiment sweep)
0.003-0.006 BPB
P2 / Focal Loss (gamma=2)
Difficulty-aware: (1-p)^2 * (-log p) down-weights easy tokens, focuses gradient on hard tokens. Rarely used in LLM training — conventional wisdom says it hurts calibration. But for BPB optimization, hard tokens dominate the loss.
Causal depthwise conv1d for cheap local context mixing. Frees attention capacity for long-range. ~3K params/layer for kernel=4. From ConvMixer/Conformer/Mamba lineage.
New technique: Over-Encoding (OE) — Keep Scylla's 998 output vocab but use hierarchical n-gram input embeddings (sum of 1-gram + 2-gram + 3-gram tables). Creates exponentially larger effective input vocab with <5% memory overhead. Log-linear loss reduction reported. Could be the key to sub-0.9.
SLOT scaling insight: More optimization steps (8→32) are nearly free since features are cached. Rank-4 delta (2048 params) and multi-layer delta (layers 9-11) are unexplored extensions with medium-high upside.
What Doesn't Work
Three failure patterns. (1) Throughput cost exceeds quality gain. In a 600s budget, anything adding >10% step overhead needs >10% per-step improvement to break even. QAT (#236: 115ms vs 67ms baseline), NorMuon (#236: 110ms), and MTP (#212, #236: 86ms) all fail this test. Partial reversal:#1031 uses MTP as auxiliary-only training signal (2-head, weight=0.1, discarded at export) with -0.0037 BPP claimed at zero artifact/eval cost — a different usage pattern (1 seed, unvalidated). (2) Mechanism redundancy. Stacking two techniques that extract the same signal yields diminishing returns — TTT+XSA underperforms XSA-alone (#290 vs #265), error-guided TTT doesn't improve over uniform TTT (#296), EMA without XSA hurts (#201). (3) Regime incompatibility. Techniques optimized for int6 break under different weight representations — the standard stack (XSA, SmearGate, WD, EMA/SWA, TTT) all fail on ternary (#367), and recurrence amplifies quantization error 900× (#363).
Late QAT at 12L is step-budget-dependent.@saml212's #332 found that at 12L, Late QAT added ~7ms/step overhead, costing ~770 training steps. At 11L, those steps would cost ~7ms each — but at 12L, each step is already more expensive and step count is already lower, so the overhead-to-gain ratio worsens. Result: Late QAT was dropped from the 12L submission. Takeaway: the same technique's cost-benefit flips depending on step time and total step count. Always re-evaluate overhead techniques when changing layer count.
Advanced quantization algorithms at int6 (Non-record: Negative results — quantization algorithms, TTT, architecture, and self-generated GPTQ calibration study #756, @abaybektursun, SOTA holder). Qronos iterative Hessian (+0.0007 worse) and CDQuant coordinate descent (+0.0005 worse) both fail to improve over standard GPTQ at int6. Reason: the int6 quant gap is only +0.0036 BPB — most weights are already at their optimal grid point. At int6, GPTQ is near-optimal. Also: TTT alone is marginal on val-calibrated GPTQ stacks — 25 total failed TTT attempts across two stacks (full, MLP-down, MLP-all: all +0.0001 or neutral). Val-calibrated GPTQ rounding decisions are disrupted by gradient updates.
MoE at small scale — Non-record: MoE exploration + multi-bit quantization analysis #480: 2-expert soft-routing MoE = −0.06 to −0.08 BPB vs dense baseline on 8xH100. Apple scaling laws (ICML 2025) confirm optimal sparsity = 0 below ~500M params. MoE is definitively unviable at competition scale.
Systematic frontier negatives (Non-record: Negative results & insights from 24hrs on 8xH100 #375, 13 techniques on Record: 11L Partial RoPE + LN Scale + EMA + XSA4 (val_bpb: 1.1248) #315 base, $500/24hrs on 8xH100).Reptile meta-TTT: +0.0076 worse (1.1332, 20% training budget consumed). All 3 TTT variants (low-LR, high-LR MLP-only, Reptile) failed. Also failed: memory tokens (+0.016), Canon layers (48% overhead), MTP (+0.028), gradient-guided quant (noise), cautious WD (breaks torch.compile: 710× slower), label smoothing, L1 reg, 1M batch, full-run QAT. Key positive findings: EMA > standard SWA by 0.003 (3-seed); 786K > 524K by 0.004 (total tokens > gradient steps at frontier). Throughput heuristic: each 1ms step overhead ≈ 0.006 BPB — any technique adding Nms must deliver >N×0.006 to break even. INT4 quant gap: 0.048-0.060 (exponential from int6's 0.006). 3-seed std: 0.0007 BPB.
INT4 quantization — catastrophic. Non-record: MoE exploration + multi-bit quantization analysis #480's controlled grid: int6→int5 MLP costs +0.007, but int6→int4 MLP costs +0.065 (10× worse). Full grid: attn6/mlp4 = 1.2111, attn5/mlp4 = 1.2183 vs baseline 1.1456. Int4 is a dead end.
MLA (Multi-Head Latent Attention) — [Non-record] MLA + SmearGate + BigramHash + SWA — pre-quant 1.2838 bpb #354's kv_rank=128 MLA on 13L runs at 83ms/step vs ~43ms baseline, halving token throughput (~3.7B vs ~7.2B tokens). Pre-quant 1.2838 — architecture quality may be there but throughput cost makes it infeasible in the 600s budget.
FTLE per-row precision (Non-record: 12L Low-Rank Q + QAT (1xH100, pre-quant 1.2035) #316) — Dynamical systems-inspired row-level quantization (Lyapunov exponent tracking). Clean negative result: uniform int-N beats FTLE-guided mixed precision at every bit width, because mixing bit widths within a row increases quantized value entropy, which defeats zstd compression. Lower RMSE does not imply smaller artifact.
Non-record: 11L XSA-all + Full GPTQ + Selective Pruning (val_bpb=1.1154, 3-seed) #609 frontier ablations (16 techniques on Record: Full GPTQ + LeakyReLU² + Parallel Muon + BigramHash 3072 (val_bpb 1.1163, 3-seed mean) #593 stack). On the current best non-TTT base: VRL +0.0012 (conflicts with VE128), Gated Attention +0.0011 (3% step overhead), Catalytic Residuals −0.0001 (redundant with existing scaling), Backout −0.0005 (redundant with U-Net skips), TrigramHash +0.0049 (hurts compression), Hadamard rotation −0.0002 but +0.5MB (net negative for artifact), Temperature scaling +0.0002 (model well-calibrated at T=1.0), seq4096 eval catastrophic (RoPE breaks beyond training length), lzma at 99.7% Shannon limit (entropy coding gains capped at 0.05MB).
Knowledge Distillation — dead at this budget (#1029). Hard distillation catastrophic (+0.090 to +0.407 BPP). Soft KL (top-32 cached logits from 105M teacher): +0.003 BPP worse at all alpha values. ~11ms/step I/O overhead costs ~280 steps; knowledge transfer doesn't compensate. Extended 2-hour training shows no crossover — curves track identically. Online distillation (761ms/step) is a non-starter. At 600s budgets, per-step overhead is fatal.
Compression moonshots — MSE ≠ artifact size (#1048, 8 findings). Procrustes symmetry-transport: 91% MSE reduction but 380% larger artifact (rotation matrices are dense/high-entropy). Low-rank rotation: rank-128 captures only 16.6% of variance. 3% pruning increases artifact by 728KB (zeroing hurts zstd-22). Takeaway: int6 + zstd-22 is near-optimal. Always measure compressed artifact, not RMSE.
MC Dropout ensembling (#1021) — K=16 passes at dropout=0.30: +0.005 BPP. dropout=0.05: +0.002. Sub-networks lack diversity at 17M params. Deterministic single pass strictly better.
SelfExtend / extended context 4096 (#1103). +0.48 BPP. The model was trained at seq2048; extending to 4096 at eval time causes massive degradation.
Mixed-precision GPTQ int4 attn / int8 MLP (#1103). +0.047 BPP. Int4 attention weights lose too much information. Hessian sensitivity says MLP is more important, but int8 MLP still can't compensate for int4 attention.
2:4 Structured Sparsity (#1105): +0.672 BPB. Definitively dead at competition scale.
Turbo-Muon on 8xH100 (#1105): +0.0018 BPP worse AND artifact over 16MB. Early convergence advantage at step 500 doesn't hold at 7000+ steps.
SLOT causality concern (#1105): @abaybektursun found shared delta optimized over all positions then applied to all positions leaks future tokens. Removed from their submission. However, #1176 and #1172 use SLOT successfully — implementation details may differ.
1xH100 not a viable proxy (#1186): 8x fewer optimizer steps means results don't transfer to 8xH100.
SGD+momentum TTT (#1186): +0.065 BPP vs AdamW. Use AdamW for TTT.
Scale deception (#1227): Local experiments can be 180° wrong at full scale. SSM hybrid showed -18% CE improvement at dim=192 but was +2.7% BPP worse at dim=512 on H100. Systematic bias, not noise. Takeaway: always validate on the target hardware and model size.
Product Quantization (#1227): +292% BPP. Catastrophic at competition scale.
PAQ Logistic Mixing (#1227): BPC=19. Fundamentally broken for multi-class prediction — compression-domain techniques don't transfer to neural LM output heads.
Complementary Training (#1227): +2.6% BPP at full scale. Does not help.
QAT as regularizer (#1227, positive): Quantized model beats float32 by 0.66%. QAT can improve BPP, not just preserve it — acts as beneficial regularization.
The Current Baseline Stack
The foundation that most competitive submissions share. Worth noting: several top submissions diverge from consensus in specific ways that paid off — #180 used int5 (former official SOTA), #236 used 524K batch instead of 786K, #76 dropped QAT and raised LR, #265 added XSA from a recent paper. The meta is a strong starting point, but the data shows room to improve individual components.
The core five: Integer quantization (int6-all or int5-MLP/int6-attn) + MLP 3x expansion + sliding window eval (stride=64) + zstd-22 compression + precision passthrough for sensitive layers (usually FP16 tied embedding; #236 uses int8 to fund MLP capacity). Near-universal across all competitive submissions, though quant precision varies — #76, #267, and former SOTA #180 use int5-MLP to fund larger BigramHash or extra layers.
Near-consensus optimizer settings: Muon momentum 0.99 (warmup from 0.92 over 1500 steps), halved LRs (matrix=0.02, scalar=0.02, embed=0.03), warmdown 3000 iters, grad clip 0.3. Most top submissions use these. Exceptions: @unixmadtoonslab's #76 (1.1468) uses higher LRs (0.03) and lower momentum (0.97). @saml212's #236 (1.1400) used 524K batch instead of 786K, gaining 0.017 BPB via more gradient updates. However, #375's systematic study on the #315 frontier base found 786K > 524K by 0.004 BPB (3-seed) — at the frontier, total tokens matter more than gradient steps. The optimal batch size is stack-dependent: 524K helps Tier 2-3 stacks; 786K helps XSA+EMA frontier stacks.
Part of the top stack: SmearGate + BigramHash + OrthoInit — used by most top validated entries. Requires OrthoInit to work (per #212's ablation). 11 layers + WD 0.04 + weight averaging (SWA or EMA). The standard-arch frontier (#414, 1.1228) builds on EMA + XSA4 + GPTQ-lite + Tight SWA + VE128 + Partial RoPE + LN Scale + Late QAT. The overall non-TTT frontier is now #609 (1.1154, XSA-all + Full GPTQ + Selective Pruning + Parallel Muon, @saml212).
Common but not universal: QAT with STE (~half), SWA (~17/49 validated), NorMuon (~3/49), FA3 (~13/49).
The Core Five Explained (for newcomers)
1. Int6 Quantization (instead of Int8)
Standard post-training quantization maps each weight to an 8-bit integer (256 levels). Int6 uses only 6 bits (64 levels, range [-32, 31]) with per-row scale factors, then compresses with zstd (level 22) instead of the baseline's zlib-9. Int6 frees ~25% more artifact space than int8, reinvested in a bigger model. Some submissions keep sensitive layers in fp16 (tied embedding) or int8 (embeddings) to limit compounding precision loss.
Origin:@nanlliu introduced int6 mixed precision in #39.
2. MLP 3x Expansion
The baseline uses 2x MLP expansion (hidden dim 1024 for 512-dim model). Top submissions use 3x (1536). Wider MLP = more expressive capacity, funded by int6 artifact savings.
Origin:@jfprincz in #70 (Mar 19 08:57 UTC). @saml212 independently reached the same insight in #61 later that day.
3. Sliding Window Evaluation
Overlapping windows (stride=64, window=2048) give each scored token 1984+ tokens of context vs minimal context with non-overlapping chunks. Purely eval-time. Worth 0.034 BPB per @samacqua's ablation in #77.
Origin:@mattqlf in #50. Stride debate: stride=256 gives marginally better BPB at 4x less eval time (#114). Doc isolation hurts at stride=64 — use flat-stream eval (#199).
4. FP16 Tied Embedding
The tied embedding matrix (input + output) is uniquely sensitive to quantization — errors compound in both directions. Keeping it in fp16 (~1MB) is the single highest-value precision decision.
Zstandard at level 22 squeezes int6 data significantly tighter than zlib-9 — enough to fit ~1-2M more parameters. Compression happens once after training; decompression is fast. Free lunch.
The Path Down: What Separates Each Tier
Post-enforcement (Mar 27), the competition has bifurcated into two tracks: n-gram cache + neural base (record submissions from 0.44 to 1.05 BPB, though many face compliance scrutiny) and pure neural frontier (1.05-1.12 BPB). Official SOTA is now 1.1147 (#1019). Novel eval-time methods like TARA (#1055, closed for causality violation) and DeltaNet Crawler (#1047, 0.8822, causality concerns flagged) explored new directions but have not yet produced compliant results.
Tier 1: Tweaking the Baseline (1.20–1.22 BPB)
Submissions in this range make one or two changes to the baseline: a longer sequence length, a learning rate sweep, a warmdown adjustment. The approach is "how do I improve this model?" — treating the baseline as mostly correct and looking for low-hanging fruit.
This works for the first 0.02 BPB, but hits a wall fast. The constraint isn't hyperparameters — it's the artifact budget. At int8+zlib, you can't fit enough model capacity to go further. Many submissions in this range are also on non-standard hardware (RTX 4090, Apple Silicon, 1xH100), which limits training tokens and disqualifies from the record track.
What to do if you're here: Adopt the core five (int6, MLP 3x, sliding window, FP16 embed, zstd-22) as a package. Each technique is well-documented in the deep dives below. Together they're worth ~0.05-0.07 BPB — the single biggest jump available.
Tier 2: Stacking Known Techniques (1.15–1.18 BPB)
These submissions adopted the core five and are assembling additional techniques: SmearGate, BigramHash, SWA, QAT, NorMuon. The approach is "what techniques exist and how do I combine them?" — surveying PRs, identifying high-impact components, and building a combined recipe.
This is effective: the leap from 1.22 to 1.16 is largely a stacking exercise. But submissions in this range often stop at "I added all the techniques" without investigating interactions. Common patterns: using SmearGate without OrthoInit (which hurts — per #212's ablation), running QAT from the start (which hurts — late QAT at 70-85% is better), or using SWA without sufficient weight decay (SWA shows no effect below WD=0.04).
What to do if you're here: Run ablations. Remove one technique at a time and measure the delta. You'll often find that one "improvement" is actually hurting because of interaction effects. Check your hyperparameters against the consensus (LR=0.02, momentum=0.99, warmdown=3000) but also against divergent successes like #76 (LR=0.03, momentum=0.97). Multi-seed validation (3 seeds) is essential — single-seed scores can be off by 0.002+ BPB.
These submissions adopted the full technique stack and understood why each technique works. @jfprincz (#198 at 1.1326) is the canonical example: 11 layers + SmearGate + BigramHash + OrthoInit + WD 0.04 + SWA + FA3 assembled into a coherent system where each piece reinforces the others — WD makes weights compressible AND quantization-friendly, SmearGate+OrthoInit inject bigram context the small model can't learn from attention alone, and SWA smooths the weight landscape during warmdown.
The approach is "how do these techniques interact, and what's the optimal system?" Key markers of Tier 3 thinking:
Ablation-driven development — every addition is measured, not assumed helpful
Precision budgeting — spending fp16 only where quantization error hurts most (tied embedding, late-layer keys)
What to do if you're here: Solidify your baseline with multi-seed validation. The primary path to Tier 4 is adopting XSA + Full GPTQ + EMA. The #609 stack (XSA-all + Full GPTQ + Selective Pruning + Parallel Muon + LeakyReLU²) reached 1.1154 but is non-record due to eval-time GPTQ — the techniques are valid if GPTQ calibration is moved into the 600s training budget. The official record SOTA target is #1019 at 1.1147. XSA + EMA is the shared infrastructure across all frontier submissions.
Tier 4: Pure Neural Frontier (<~1.120 BPB)
The official record SOTA is #1019 at 1.1147 (@abaybektursun). The #609 stack reached 1.1154 (non-record due to eval-time GPTQ) and GEPA #505 reached 1.1181 (artifact >16MB). Both demonstrate what's achievable with compliant implementations.
The key insight at Tier 4: EMA (0.997) outperforms standard SWA by 0.003 BPB (#375, 3-seed verified). #315 demonstrates that the XSA+EMA base still had headroom via careful regularization — Partial RoPE, LN Scale, and Late QAT each target a specific weakness.
What to do if you're here: Three options. (a) Beat #549 on pure neural: Adopt the #609 technique stack with GPTQ calibration inside 600s training budget. Remaining untried: Mousse optimizer, OptRot, systems opts (Liger-Kernel, 2:4 sparsity). (b) Add n-gram cache (→ Tier 5): The single biggest lever — 0.07-0.16 BPB from a legal backward-looking n-gram eval cache. (c) Legal TTT with compliant GPTQ: All frontier TTT submissions were closed for eval-time GPTQ. The recipe works if GPTQ calibration fits in 600s. GEPA + legal TTT at 1.0983 on 4xA100 (#628, 20k steps) — 8xH100 version untried.
Tier 5: N-gram Cache (Invalidated Mar 27)
The n-gram cache track was invalidated after discovery that hashed implementations scored only the correct token without full-vocabulary normalization. #978 proved correctly normalized n-gram achieves only 1.51 BPP (worse than neural baseline). 33+ PRs closed by @valerio-oai. Whether a correctly normalized eval-time statistical method can improve on pure neural remains an open question.
Technique Interactions Matter More Than Technique Count
A recurring pattern: techniques that work independently can fail in combination. TTT+XSA actively hurts (#303: +0.016 worse), EMA fails without XSA (#201) but succeeds with it (#287), and 12L fails at seq2048 but works at seq1024 (#219 vs #76). #474 confirms this extends to newer techniques: VRL + Gated Attention + Catalytic Residuals stacked on a 12L SWA base (no XSA, no EMA) yielded 1.1690 — worse than the same base without them (1.1466). Frontier techniques are optimized for the frontier base; applying them to weaker bases produces negative or null returns.
The untried combinations above should be evaluated against your specific model's weaknesses, not applied blindly. XSA + EMA appears to be a prerequisite for most newer techniques (VRL, GA). For the pure neural track, the strongest remaining candidates are systems optimizations (fused kernels, 2:4 sparsity — throughput gains with significance waived) and compression innovations (OptRot, entropy-coded weights). For the overall frontier, n-gram eval cache is by far the highest-impact lever available.
TTT ruling (Mar 20, @0hq on #152): Only backward-looking TTT allowed — adapt on tokens already graded, not future tokens. Pre-eval adaptation invalid. Causal TTT (#267-style) remains allowed. In README FAQ.
Mar 22, @cocohearts on #317: TTT is "not in the spirit of the challenge." Broader organizer signal — even backward-looking TTT may face scrutiny.
Mar 23, @0hq on #402: Explicit TTT clarification — token-stream model is correct. You may use any preceding eval tokens already graded. You may NOT re-order the evaluation set. Invalid TTT PRs (train-on-val-then-measure) will be closed. Auto-review process being built.
Mar 23, @cocohearts:#374 rejected for insufficient statistical significance vs new SOTA. #505 needs packaging fixes.
Mar 24, @valerio-oai — enforcement sweep (15+ PRs closed). Two categories: (1) TTT information leakage: multi-epoch TTT with min-NLL selection, and adapting-then-scoring same tokens, both ruled equivalent to "training on the val set." #593, #576, #573, #568, #596, #605, #614, #620, #518, #548 closed. (2) Training data at eval time: GPTQ calibration using training data during eval budget disallowed. #593, #576, #569 closed for this. Calibration must count within training 600s. #589 ruled valid but closed — fails 0.005-nat threshold vs #549 SOTA. valerio-oai confirmed: "TTT is a valid approach in theory" but "very easy to unintentionally leak val data into."
Mar 25, @valerio-oai — second enforcement sweep (issue #677). Comprehensive audit. (1) Eval-time GPTQ: Training for full 600s then doing GPTQ calibration afterward (even 3-4s) is "accessing training data at eval time" — disallowed. #606, #615, #626, #639, #656 closed.(2) N-gram eval cache ruling: The concept is "directionally legal" — building a cache from already-scored tokens is allowed. The specific #659 implementation was illegal (hindsight selection: comparing n-gram vs LM on the true next token). Legal alternatives: fixed-weight blending or entropy-adaptive alpha (using model uncertainty, not ground truth). (3) #706 flagged:@valerio-oai told @newjordan that #706's GPTQ calibration still runs after 600s training time — needs fix. (4) Broad invalid TTT list:#410, #415, #417, #442, #462, #481, #486, #517, #518, #532, #555, #581, #595 all flagged for adapting on validation before the reported eval pass.
Mar 27, @valerio-oai — mass n-gram cache closure (33+ PRs, #677). Hashed n-gram caches disallowed: they score only the correct token via hashing without normalizing over the full token distribution, producing invalid probabilities. Two-pass rescoring (score → TTT → rescore) explicitly disallowed as "training on the eval set." PRs closed include #846, #853, #868, #869, #870, #876, #881, #888, #893, #900, #907, #912, #918, #982, and many more. Only reviewing PRs after #988 for potential merging. Recommended: @NoesisGenesis's formal criteria for valid causal prediction — (a) distribution depends only on artifact + strict prefix, (b) full normalized distribution over token vocabulary required before scoring, (c) score computed from pre-update probability only, (d) single left-to-right pass.
Mar 28, @valerio-oai on #728: Val-calibrated GPTQ "breaks autoregressivity" — disallowed. Self-generated calibration data (as in #1019) is "probably legal." PR left open pending fix.
Mar 28, @valerio-oai on #991: Closed for double-pass TTT (score, train, rescore same tokens).
Mar 28, @valerio-oai on #1028: GPTQ calibration running after 600s training cap = accessing training data at eval time. Author confirmed bug and resubmitted as #1047 with GPTQ within budget. Under organizer review.
Technique Deep Dives
The Muon Optimizer Family
Muon (MomentUm Orthogonalized by Newton-Schulz) is the optimizer at the heart of this competition's baseline, created by Keller Jordan for the NanoGPT speedrun. It runs standard SGD with Nesterov momentum, then post-processes each 2D parameter's gradient update by replacing it with the nearest orthogonal matrix via Newton-Schulz iteration. Intuitively: compute the gradient direction, then "clean it up" so the update is maximally informative without redundant directions. It's equivalent to steepest descent under the spectral norm, which improves the conditioning of the optimization landscape. ~35% faster training than AdamW on language models.
NorMuon extends Muon by adding per-neuron adaptive learning rates from accumulated second-order statistics. Vanilla Muon can produce updates with highly non-uniform norms across neurons, causing some neurons to dominate training. NorMuon normalizes row-wise after orthogonalization, combining Muon's conditioning benefits with Adam-style balanced per-neuron learning. It also improves distributed scaling by avoiding full momentum gathering across GPUs. Used by @mtybadger (#122), @vmfunc (#89), @abhishekgahlot2 (#137), and others.
Muon Weight Decay — The competition baseline's Muon optimizer has no weight decay. Decoupled weight decay for Muon (p.mul_(1 - wd * lr)) existed in modded-nanogpt since Nov 2025, but wasn't in the baseline. @notapplica was the first to bring it into this competition in #60, improving BPB from 1.2160 to 1.2094. Weights stay smaller and better-distributed, improving both generalization and compressibility.
Post-enforcement status (Mar 27): The Mar 27 enforcement sweep closed 33+ n-gram cache PRs after discovery of the normalization bug — implementations scored only the correct token without full-vocabulary normalization, producing artificially low BPP. #978 proved that properly normalized n-gram achieves only 1.51 BPB (worse than neural baseline). The question of whether a correctly normalized eval-time statistical method can improve on pure neural remains open.
Quantization-Aware Training (QAT) with STE
Instead of training in full precision and quantizing afterward, QAT simulates quantization during training. In the forward pass, weights are rounded to their quantized values. The problem: rounding is non-differentiable, so gradients can't flow through it.
The Straight-Through Estimator (STE) solves this by pretending the rounding operation is the identity function during the backward pass. It's mathematically "wrong" but works remarkably well — the model learns weight configurations that are robust to precision loss because it's been "seeing" quantized weights throughout training.
Late QAT outperforms full-training QAT: The later, the better. @trovatochris (#117) activates at 70%, @mohosy (#130) at 75%, @unixmadtoonslab (#76) at 85%. #76 even dropped QAT entirely at 12L (1.1468), finding WD=0.04 alone sufficient. @jfprincz's #315 pushes this to the extreme: STE activates only in the final 4% of training (lr_scale < 0.1, during low-LR warmdown). This cuts the int6 roundtrip gap to ~0.007 BPB while preserving full-precision convergence. The lesson: QAT activation is a spectrum — later = cleaner convergence, better int6 gap.
Int8 vs int6 QAT tradeoff:@mrdavtan's ablation in #145 shows that int8 QAT is not worth it under the 10-min wallclock cap. The torch.quantile call for exact percentile matching adds ~20% per-step overhead (64ms → 77ms), costing ~2,000 training steps. Result: 1.2052 BPB with QAT vs 1.1925 without — the lost training tokens hurt more than closing the ~0.007 int8 quantization gap. Int6 QAT, however, likely pays off because its larger ~0.01+ BPB gap justifies the overhead — confirmed by #128 and #137.
SmearGate & Bigram Hash Embedding
@unnir introduced SmearGate in #102 and refined it in #135. This appears to be a novel technique for this competition — no published papers found.
SmearGate: A tiny learned gate (~512 params) that blends each token's embedding with the previous token's. This injects bigram (two-token) context directly into the embedding layer before the transformer starts processing. Normally a transformer must discover token pair relationships through self-attention; SmearGate provides this signal for free.
Bigram Hash: A hash table (commonly 2048-10240 buckets, dim=128, projected to 512) that maps token pairs to learned embeddings. Together with SmearGate, this gives the model token-pair awareness at nearly zero parameter cost.
@unnir's original combination with orthogonal initialization achieved 1.1539 BPB in #135. @jfprincz's #198 (1.1326) extended this with 11L + SWA + FA3 + WD 0.04, and #287 (1.1280) extended further with XSA + EMA.
OrthoInit appears critical for SmearGate.@mrdavtan's ablation in #212 found that adding SmearGate + BigramHash without OrthoInit hurt BPB (1.1739 vs 1.1708 without). Every successful SmearGate submission uses OrthoInit — the two techniques may be co-dependent.
Exclusive Self-Attention (XSA)
XSA (arXiv:2603.09078, Shuangfei Zhai, 2026) removes self-value bias from attention output via orthogonal projection. In standard attention, each token's value vector contributes to its own output — XSA subtracts this self-component, forcing the model to rely on information from other tokens. Applied to the last 3-4 layers only ("Partial XSA"), where self-attention bias is highest.
Zero parameters, minimal overhead.@unnir's #265 GQA-aware implementation reduces XSA overhead from ~7ms/step to ~2ms/step. Near-universal among frontier submissions. Best non-TTT (#609, 1.1154) uses XSA on all 11 layers; official SOTA (#1019, 1.1147) uses XSA-all.
XSA coverage depth: 4 layers appears near-optimal.@gowtham0992's #478 tested XSA on ALL 11 layers: 1.1268 (3-seed) vs XSA-4 at 1.1327 on the same base (−0.006 from XSA-all). But #414 (XSA-4 + VE128 + Partial RoPE + LN Scale) reaches 1.1228 — better than #478's XSA-all(11) at 1.1268. XSA-all adds ~3ms/step overhead (−230 steps), and removing self-value from ALL layers may degrade the model's own-representation capacity. The progression: 3 layers (#265: 1.1307) → 4 layers (#414: 1.1228) → 11 layers (#478: 1.1268) suggests 4-6 layers is the sweet spot for non-TTT. However, #609 (1.1154, best non-TTT) uses XSA-all(11) and #606 (1.1162, best legal TTT) also uses XSA-all — at the current frontier, XSA-all with Full GPTQ overcomes the overhead penalty.
Test-Time Training (TTT)
@samacqua introduced a creative approach in #77: adapting the model during evaluation.
For each validation document, rank-8 LoRA (Low-Rank Adaptation) adapters are trained on the document's own text using only backward-looking context (no data leakage). The model essentially "studies" each document briefly before being scored on it. LoRA makes this practical by only training tiny low-rank matrices (~1.5% of params) rather than the full model, enabling batched per-document adaptation within the eval time budget.
Original #77 ablation showed TTT itself adds ~0.003 BPB on early baselines (most gain came from doc isolation + sliding window). Full-model SGD TTT (#152) was ruled invalid by @0hq — only backward-looking (score-first) TTT is legal. The best legal TTT submissions (#606 at 1.1162, #615 at 1.1169) were later closed for eval-time GPTQ on training data (Mar 25 sweep).
TTT on XSA+EMA is a spectrum, not a binary. On SmearGate bases: #254 shows 0.014 BPB gain. Three XSA+EMA data points, sorted by base strength: (1) #317 (weak base, pre-quant 1.1581, no FA3): TTT gains 0.024 BPB. (2) #338 (@alertcat, #315 base — frontier at 1.1250, Partial RoPE + LN Scale + Late QAT): TTT neutral ±0.001 (3 seeds). (3) #303 (@sseanliu, #287 base — 1.1280, without #315's additional regularization): TTT +0.016 BPB worse. The pattern suggests TTT interacts with how tightly converged the base model is: under-trained bases benefit from local adaptation; over-regularized frontier bases are disrupted; the current frontier (#315) sits in a neutral zone. #338's neutral result is informative — it means TTT is not a meaningful lever at the frontier.
Reptile meta-TTT: gains on SmearGate, fails at frontier.@sseanliu's #296 shows 0.011 BPB on SmearGate models vs 0.001 naive. But #375 tested Reptile on #315's XSA+EMA base: +0.0076 worse, consuming 20% of training budget. The SmearGate gain does not transfer to the frontier. All three TTT variants (naive, MLP-only, Reptile) are now confirmed dead ends at ~1.125. Error-guided TTT is also negative — hardest tokens are genuinely unpredictable.
TTT optimizer recipe matters.@Christopher-Lee-McClendon's #461 (non-record, 4xA100) found that SGD+momentum(0.9), 3 epochs per 32K chunk, freezing first 2 blocks gets −0.0165 BPB TTT gain — 2.4× better than AdamW 1-epoch over all params (−0.0068 in their prior #456). Pre-TTT baselines nearly identical, so the entire improvement comes from the TTT recipe. This partially contradicts the #442 narrative (AdamW >> SGD) — the comparison is more nuanced: selective freezing + multi-epoch SGD with momentum can outperform single-epoch full-network AdamW.
Legal TTT survivors — none remain after Mar 25 sweep.#606 (1.1162) and #615 (1.1169) were closed — eval-time GPTQ calibration on training data. #576 (1.1164) closed in Mar 24 sweep. #573 (Multi-Pass min(NLL)) ruled invalid. All frontier TTT submissions used eval-time GPTQ and are now invalid. TTT optimizer matters for GPTQ: SGD TTT hurts Full GPTQ models (+0.030, #601), but AdamW with cosine LR works. The remaining legal TTT avenue requires GPTQ calibration within the 600s training budget.
Cosine TTT scheduling is a 3× multiplier.@mrdavtan's #481 (3-seed, 1.0970) introduced two TTT innovations on top of AdamW TTT: (1) cosine LR decay over 30 epochs — high LR early to repair quant damage, low LR late to refine; (2) per-layer LR groups based on measured quantization error — 3× base LR for MLP output projections (3.4× higher quant error), 0.5× for input projections. Result: TTT gain of −0.061 BPB vs #442's −0.019 with flat LR — a 3× improvement from scheduling alone. Pre-TTT ~1.158 (weaker base, FA2 not FA3). Also tested: focal loss and KL-divergence from pre-quant model — both failed to improve over CE. ⚠️ Pre-eval TTT.
N-gram Eval Cache (the Mar 25 revolution)
The single biggest BPB lever discovered in the competition. During sliding window evaluation, a backward-looking n-gram cache is built from already-scored tokens and mixed with model predictions. The concept is simple: if the model has already scored "the cat sat on the", and the 5-gram "cat sat on the" was followed by "mat" last time, weight that prediction into the next token's distribution.
How it works:
After scoring each token, record the preceding n-gram context and the actual next byte
For new tokens, look up the current n-gram context in the cache
If found, mix the empirical n-gram distribution with the model's distribution: p_final = (1-α) * p_model + α * p_ngram
Store counts in a hash table (count-min sketch, ~4M buckets)
Why it's so effective: Language has enormous local repetition — names, technical terms, formatting patterns — that a small transformer can't memorize but n-grams capture perfectly. The n-gram cache acts as a lossless "local memory" that costs zero artifact bytes (built on-the-fly from eval data).
Legality argument: All implementations claim score-first backward-looking compliance — the cache uses only previously-scored tokens, alpha depends on the model's own entropy (not ground truth), and there's no oracle selection. #702 cites @valerio-oai suggesting entropy-adaptive alpha as a legal alternative in the #659 review. But the technique hasn't been officially ruled on yet.
#738 adds kNN-LM:@gowtham0992 stores 512-dim hidden states in a GPU ring buffer and finds k=32 nearest neighbors for uncertain tokens. RBF kernel builds a non-parametric distribution. Additive −0.007 BPB on top of n-gram cache. Based on Khandelwal et al. 2019 (ICLR 2020). Captures semantic patterns that pure n-gram statistics miss.
#315's Techniques: Partial RoPE, LN Scale (Late QAT was inactive)
@jfprincz's #315 (1.1250) adds two effective zero-parameter techniques on top of #287's XSA+EMA base, gaining 0.0023 BPB. Note: Late QAT was also included in the code, but torch.compile constant-folded the _qat_enabled flag, making the STE branch dead code — Late QAT never activated (discovered by @152334H, confirmed in #453). Update:@wfproc (#1032) confirmed this dead-code bug persists in the current SOTA #549 codebase. A fix via tensor-scale STE actually worsened the int6 gap — suggesting WD+EMA already compensate for what QAT was supposed to do. The 0.0023 gain comes entirely from Partial RoPE + LN Scale.
Partial RoPE (16 of 64 head dimensions). Rotary Position Embedding (RoPE) injects position information by rotating query/key vectors. Standard RoPE applies to all head dimensions. Partial RoPE applies to only 25% (16 of 64 dims) — the remaining 48 dims attend without position encoding. Why this helps: the position-free dims learn semantic similarity independent of token distance, improving generalization across different position ranges. The model can learn both "what things are" (position-free) and "where things are" (position-encoded) using different parts of the same head. Zero new parameters.
LN Scale (output scaled by 1/√(layer_idx+1)). After each RMSNorm, the output is multiplied by a layer-dependent scale factor that shrinks with depth. Layer 0: ×1.0; Layer 5: ×0.408; Layer 10: ×0.302. This damps the contribution of deeper layers to the residual stream, preventing later layers from "overwriting" early representations. Training is more stable — the model can use depth incrementally rather than being forced to route everything through deep layers. The 1/√(layer+1) schedule is related to the "depth scaling" used in some architecture papers. Zero new parameters.
Late QAT (STE enabled only when lr_scale < 0.1) — ⚠️ was dead code in #315.torch.compile constant-folded the _qat_enabled class attribute, so the STE branch never activated (discovered by @152334H, confirmed in #453). The concept is sound — late activation avoids corrupting Muon's momentum — but #315's actual gains came from Partial RoPE + LN Scale alone. Working Late QAT:@unnir (#374, scale<0.1), @signalrush (#414, threshold 0.15), @fbedev (#417). Downstream submissions copying #315's code may also have inactive Late QAT.
The two active techniques (Partial RoPE + LN Scale) gain 0.0023 BPB vs #287 — statistically clear (3-seed variance 0.0005 BPB, t-stat -101.9 vs SOTA, p << 0.01).
Canon ACD layers (Allen-Zhu 2025) on 9L stack — learnable 1D conv (k=3) placed before attention, before MLP, and in MLP hidden stream (avoids QKV=B for cost). 1.1668, 1 seed. Novel architecture technique; interesting if it scales to 11L.
12L Low-Rank Q (r=128) + QAT int7 on 1xH100 (pre-quant 1.2035, awaiting 8xH100). Key negative result: FTLE per-row precision is a dead end — uniform int-N beats mixed-row at every bit width due to higher entropy defeating zstd. Layer sharing also abandoned at 512d (costs 0.09 BPB, no space benefit).
Neural Cache research proposal — maintain per-layer KV cache across sliding windows, extending effective context from 2K to 50K+. Zero artifact cost, backward-looking compliant. Untested (torch.compile state bug). Proposed on #287 base (1.1284).
VRL+GA on 11L production stack (1xA6000, 14.5hr). 1.1720 BPB, 19.4MB (over limit). Confirms dev ablation (−0.017 additive). Not 8xH100 — VRL on 8xH100 frontier still untested by originator (#486 by @ndokutovich tested VRL+Cosine TTT at 1.0887).
First VRL+GA+Catalytic Residuals stack on 12L + BigramHash(10240) + SWA + Late QAT. 1.1690 — disappointing vs #450's 1.1466 (same base without VRL/GA). Techniques don't stack additively here: no XSA, no EMA → weak base dilutes gains.
Delayed PPM eval-time bank on #180 base. Classical n-gram backoff (C trie) with 2048-token delay — only sees tokens outside transformer's window. −0.00126 BPB (p=0.000041, 3-seed) — real but below 0.005-nat record bar. Zero artifact cost, composable with any model. First positive classical compression result at frontier.
Masked Diffusion (MDLM): First text diffusion submission (README wishlist). Bidirectional attention, pseudo-log-likelihood eval. 1.3600 BPB, 3-seed, 12.9MB. High school student's 2nd ML competition.
JEPA-LM negative result: -19.5% CE on synthetic Markov chains but only -0.24% on real text. +40% throughput overhead makes it net-negative. Valuable lesson: synthetic benchmarks don't transfer.
QAT dead-code confirmed in SOTA #549 (torch.compile constant-folds Late QAT). 7 techniques all negative. Heuristic: 1ms overhead = 0.007 BPP at 83ms/step. 1xH100 research.
@aquariouseworkman (SmearGate + BigramHash — first used in competition by @aquariouseworkman (#65, Mar 19 07:42 UTC). BigramHash separately developed by multiple authors.)
Near-universal (25+). All competitive submissions use SmearGate+BigramHash+OrthoInit.
@abaybektursun (Model autoregressively generates calibration tokens for Full Hessian GPTQ. Closes 84% of val-vs-random gap. First appeared in #728, resubmitted as #1019.)
@seanward (MTP auxiliary head (training-only, discarded at export). Introduced day 1 by @seanward (#88). Disabled in baseline (MTP_NUM_HEADS=0). Re-enabled by @michaelwinczuk (#1031, weight=0.1) with -0.0037 BPP claimed.)
@sanyalsunny111 (Training-free eval-time method: contrastive adjustment using early-layer hidden states. No gradients, no weight updates. Community flagged causality concern (scatter_ leaks target token).)
@Gusanidas (Sliding window of 512 on even layers during training, full attention on odd — 21% faster at seq_len 6144. Distinct from sliding window eval.)
@resouer (SLOT delta optimized separately per batch element [bsz,1,512] rather than shared [1,1,512]. Combined with Scored-Position mask and Logit Bias.)
@resouer (Real training data (256 batches) for GPTQ calibration within 600s budget, distinct from #1019's AR self-generated approach)
Predictions & Commentary
SLOT is now mainstream. Nine record submissions use SLOT variants. Context-Only SLOT (#1217) proved causal with only −0.0002 BPP cost. Scored-Position SLOT (#1229) pushed to 0.9300 BPP with per-sample deltas. Expect SLOT in every competitive submission going forward.
Scylla + SLOT remains the obvious untried combination.#1184 (0.9485, Scylla alone) and #1229 (0.9300, SLOT alone) are the two best pending submissions. Combining them could push toward 0.92–0.93. Someone will try this soon.
Two separate frontiers have emerged. Standard-tokenizer submissions top out around 1.0979 (#1218, simplification approach). Custom tokenizers (Scylla) dominate overall at 0.9485. These are effectively different competitions now.
The official leaderboard is about to be disrupted. The gap between official SOTA (1.1147) and top pending (#1229, 0.9300) is 0.185 BPP. When these merge, most current "competitive" submissions will fall below the new 0.005-nat threshold. Expect a wave of invalidations.
Fused kernels help but aren't the moat. CUTLASS fusion gives ~2–3% step speedup, but custom tokenizers and eval-time adaptation (SLOT, TTT) dominate the gains. Systems optimization is necessary but not sufficient.
Simplification won the standard-tokenizer race. #1218 proved that removing TTT/QAT/hash embeddings and adding 4096-vocab + higher WD (0.085) beats complex stacks. The RMS-compression insight (R²≈0.99 between weight RMS and compressed size) is a genuine contribution — expect wider adoption of WD tuning for artifact size.
Negative results are accelerating. Focal loss, self-gen GPTQ, random MLPs, oscillatory recurrence all failed this cycle. The frontier is mature enough that most "clever ideas" lose to more training steps. Time-budget tradeoffs dominate: any technique that costs >5% of training time must deliver >0.01 BPP to be net positive.
Complementary Training — tokens predictable by bigram stats get lower loss weight during training. Model specializes on what n-grams can't predict, enabling higher eval-time n-gram alpha (20-75%). + Backoff N-gram Mixer + VRL + XSA-4. Std=0.0001.
Seed-Regenerated Random Model + Incremental N-gram Cache. Model weights generated from seed (not trained) — neural baseline 1.503 BPP. All compression from n-gram cache. 1-seed only, run on MI250X (not H100). Pending H100 validation + 2 more seeds.
Coprime-Stride Loader + Full Hessian GPTQ + XSA-all. Pure neural, no TTT — sliding window only (87s eval). Coprime-stride multi-shard loader (#726-style) for batch diversity. Full Hessian GPTQ with Cholesky error compensation + column reordering (14s within training budget). XSA all 11 layers. BigramHash(2816x112). Built on #549 stack. Beats former SOTA (1.1194) by 0.012 nats. Std=0.0004.
Standard 11L SOTA stack reproduction. LeakyReLU², XSA4, Partial RoPE, LN Scale, VE128, EMA, Late QAT, Legal TTT, GPTQ-lite. Claims 1.1130 but only overall BPP reported, no per-seed breakdown visible.
MTP-2 Funnel: Multi-Token Prediction (2-head, weight=0.1) as auxiliary training signal on #549 stack. MTP heads discarded at export — zero artifact cost. LeakyReLU(0.75)².
First SLOT submission. Optimizes single delta vector (512 dims) at last hidden layer per batch during eval. Stacks on TTT: -0.0008 BPB over baseline. Also tested CTW — negative result (+0.005 worse). Based on Hu et al. arXiv:2505.12392. Std=0.0003.
Bayesian posterior packets + selective gating on #549 stack. Conjugate online updating mixes training priors with eval-time counts. +0.0006 over pure neural TTT.
Corrected full-vocab normalization rerun. With proper normalization, n-gram path degrades to 1.5134 — WORSE than neural sliding baseline. The n-gram cache 'improvement' was entirely a normalization artifact.
Parameter Golf Live AI Commentary (Inactive as of Apr. 1st)
Auto-updated every ~10 minutes. Tracking techniques, trends, idea lineage, and explaining concepts for the community.
Last updated: Apr 5, 7:56 PM PT
The Competition at a Glance
Goal: Train the best language model that fits in a 16MB artifact, training in under 10 minutes on 8xH100s. Evaluated by compression of the FineWeb validation set, measured in bits per byte (BPB) — lower is better. Tokenizer-agnostic. Baseline: 1.2244 BPB.
What does "compression" mean here?
BPB (bits per byte) measures how many bits your model needs to encode each byte of text. A model that perfectly predicts every next character needs zero bits — it already "knows" what comes next. A model with no understanding of language needs the maximum (~8 bits per byte).
A model's cross-entropy loss IS its compression rate. Shannon proved in 1948 that prediction and compression are mathematically equivalent — a model that predicts well compresses well, and vice versa. The competition measures the compression side of that equivalence.
This framing matters because it legitimizes approaches beyond pure language modeling: sliding window eval improves compression by giving more context. Backward-looking TTT adapts to already-scored tokens for better compression. These are valid compression strategies.
There is no separate held-out test set — the FineWeb validation set is the fixed evaluation target. However, val tokens cannot be stored in the artifact (paid prefix ruled out), and pre-eval adaptation on val data is also ruled out. Only backward-looking TTT (adapting on tokens already graded) is permitted.
"Tokenizer-agnostic" means BPB normalizes across tokenizers. A bigger vocabulary uses fewer tokens but more bits per token — BPB cancels that out, measuring compression of raw bytes regardless of how they're tokenized.
Record submission requirements: Artifact ≤16,000,000 bytes (code + compressed model). Training ≤10 min on 8xH100 SXM. Evaluation ≤10 min (separate budget). No network calls. New SOTA records must beat the current best by ≥0.005 nats at p < 0.01 significance (typically 3 seeds). Evaluation methods are unrestricted — any sequence length, sliding window, etc. are fair game. Test-time training is allowed only on already-evaluated tokens (backward-looking); pre-eval adaptation on val data is ruled out.
The competition launched Mar 18. The official SOTA is now 1.1147 BPB (#1019, @abaybektursun, Mar 25 — merged, AR Self-Gen GPTQ + XSA-all). The best pending submission overall is #1229 (@resouer, 0.9300 BPB, 3-seed) using Scored-Position SLOT + Per-Sample Delta + training-data GPTQ. Previously #1184 (@icryo, 0.9485) using Scylla tokenizer (998-vocab TokenMonster) + Full GPTQ + XSA-all. The best standard-tokenizer submission is #1176 (@bigbag, 1.0914, 3-seed) with QK-Gain 4.0 + Muon-TTT + SLOT. A new n-gram backoff cache submission #1185 (@skoustav35, 0.9641) claims proper Laplace normalization. @abaybektursun's #1105 (1.1125, CUTLASS EVT + MLP 3.5x + mixed int5/int6) narrowly misses the 0.005-nat record threshold vs the new SOTA. @dentity007 submitted 7 PRs addressing README wishlist items (JEPA, text diffusion, H-Net, Universal Transformer, SSM, megakernels, random linear maps) — all proof-of-concept stage. The SOTA shift from 1.1194→1.1147 means 4 former record submissions (#728, #1060, #1099, #1130) no longer meet the 0.005-nat threshold.
Blue = official leaderboard (1.2244 → 1.1147). Green = pending frontier (now invalidated n-gram submissions removed). Red dashed = official SOTA (1.1147, #1019).
Official Leaderboard (Top 5)
New official SOTA: 1.1147 (#1019, merged Mar 25). Best pending: #1184 (0.9485, Scylla tokenizer), #1185 (0.9641, n-gram backoff cache), #753 (0.9625, Podracing II). Best standard tokenizer: #1176 (1.0914, QK-Gain + TTT + SLOT). Four former records (#728, #1060, #1099, #1130) downgraded — no longer clear the 0.005-nat threshold against the new SOTA. Tables below ↓
Pending: Meets Record Requirements
Record-eligible submissions only. Pre-eval TTT entries excluded per @0hq ruling on #402 — only backward-looking (score-first, single-pass) TTT is allowed. Official SOTA: 1.1147 BPB (#1019, @abaybektursun — AR Self-Gen GPTQ + XSA-all + Legal TTT + Parallel Muon, updated Mar 24).
Top 5 record-eligible (30 total — full table in collapsible below):
Also notable: #795 (0.8881) | #788 (0.9059) | #1229 (0.9300) | #782 (0.9362) | #774 (0.9370) + 20 more
Pending: Not Yet Validated
Submissions with competitive BPB that haven't yet demonstrated statistical significance.
Top 5 not-yet-validated (100 total — full table in collapsible below):
30 record-eligible + 100 unvalidated | Official SOTA: 1.1147 (#1019) | Full tables in collapsibles below ↓
Note: The full "All Pending Validated" table below contains the pre-n-gram-cache entries. N-gram/Hedge Mixer submissions still open (#702, #715, #706, #700) and #728 is tracked in the Not Yet Validated table (downgraded when SOTA moved to 1.1147).
Untried Combinations
Ranked by expected value (likely gain times probability of working), grounded in competition ablation data:
Tier 1 — Highest expected value (n-gram cache extensions)
N-gram cache + stronger neural base. The top n-gram submissions (Record: 0.4416 BPB -- Complementary Training + Backoff N-gram Mixer #803 at 0.4416, Record: Order-Adaptive Entropy Gating + BackoffNgramMixer (val_bpb=0.5466) #798 at 0.5466) show that training-eval co-optimization is the next frontier (XSA-all, VRL, GA). Combining the best neural base (Non-record: 11L XSA-all + Full GPTQ + Selective Pruning (val_bpb=1.1154, 3-seed) #609's XSA-all + Full GPTQ + Selective Pruning stack, with GPTQ in training budget) with multi-order backoff + entropy-adaptive alpha could push below 0.30. Record: BackoffNgramMixer + Drift-Free TTT (3-seed mean val_bpb=0.6683) #779's ablation shows neural-only at 1.1363 dropping to 0.6712 with the BackoffNgramMixer alone. A stronger neural base would push even lower.
GEPA + n-gram cache. GEPA's neural-only frontier (Non-record: 11L GEPA + 20k Steps + Pure Int6 + Legal TTT (val_bpb=1.0983): unlimited compute: 4×A100-40GB, ~2.8 hours #628: 1.0983 on 4xA100) plus n-gram backoff could target sub-0.95. 8xH100 record-eligible GEPA still untried (~1.116-1.120 projected at 7k steps, pre-n-gram).
Context Tree Weighting (CTW) instead of heuristic alpha. The current top n-gram submissions use hand-tuned or entropy-adaptive alpha to mix n-gram orders. CTW (Willems et al.) provides Bayesian-optimal weighting over all context tree models up to a given depth — provably minimax optimal for tree sources. Replaces heuristic with theory. Zero artifact cost. Non-Record: SLOT Eval-Time Augmentation on PR #549 SOTA Stack val_bpb = 1.1185 (3-seed mean, std 0.0003) | ~15.9 MB | 8×H100 SXM #1084 tested depth-4 CTW: +0.005 BPB worse, 46 min eval — negative result. Original estimate questionable. Moderate complexity (tree data structure).
Logistic-domain mixing. Current submissions use linear interpolation:
alpha*p_ngram + (1-alpha)*p_neural. PAQ-style compressors use log-odds space mixing, which handles extreme probabilities better. A one-line change. Est. 0.002-0.005 BPB. Trivial complexity.Adaptive stride (entropy-guided two-pass). First pass with stride=64 scores all tokens and records per-token entropy. Second pass re-evaluates high-entropy regions with smaller stride (16-32). Targets compute where it helps most. Backward-looking, zero artifact cost. Est. 0.005-0.015 BPB. Low-moderate complexity.
Fixed-Share Hedge (non-stationary expert tracking). Record Submission: 1.0541 BPB - 5-expert Hedge Mixer + CROWN-Q + stride=64 #700's Hedge algorithm assumes stationary expert quality. Fixed-Share (Herbster & Warmuth) allows "switching" between experts — important because FineWeb contains diverse content types (code, prose, tables). Zero artifact cost. Est. 0.003-0.008 BPB over standard Hedge. Low complexity (one parameter: switching rate).
PPMII-style escape estimation (Shkarin 2001). Replace heuristic backoff with principled escape probabilities + information inheritance (new context nodes inherit counts from parent) + full exclusions (symbols already assigned probability at order-k excluded from order-(k-1)). The theoretically optimal version of what the current BackoffNgramMixer approximates. 40 years of compression research refinement. Est. 0.01-0.03 BPB. Medium complexity.
Match model (longest-match prediction). Instead of fixed high-order n-grams (8, 9, 10+), find the longest match anywhere in previously-scored data and predict based on what followed that match. Used by LPAQ/PAQ. Captures arbitrarily long repeated contexts without exponential memory cost. Complements rather than replaces multi-order backoff. Est. 0.005-0.01 BPB. Medium complexity.
Sparse/skip-gram context models. Use non-contiguous positions (e.g., tokens at -1, -3, -5) as context. Captures patterns with intervening variable content (HTML tags, code indentation, sentence structures). Multiple sparse models with different gap patterns. Zero additional memory per context — just hash different positions. Especially effective on FineWeb's structured web text. Est. 0.005-0.015 BPB. Low complexity.
Cascaded SSE stages (3-5 chained APMs). Rather than a single SSE post-processing step, chain multiple Adaptive Probability Map stages with progressively higher-order contexts. Each stage corrects residual biases from the previous one. Used by PAQ/PAQAR. Est. 0.005-0.015 BPB. Low complexity.
Complementary distillation — add
-lambda * KL(P_ngram || P_model)to the training loss. Pushes the neural model to explicitly diverge from the n-gram distribution at every token, not just binary loss reweighting. Based on "N-gram Is Back" (Li et al., EMNLP 2022) residual learning framework. Smooth, differentiable. Est. 0.01-0.03 BPB over current Complementary Training. Low complexity.Three-tier token weighting — extend Complementary Training beyond binary easy/hard. Down-weight tokens predictable by bigrams (n-gram handles them), AND down-weight "noise" tokens (random proper nouns, typos) that neither model will ever predict well. Concentrate gradient on the learnable frontier. Based on Token Weighting for Long-Range LM (NAACL 2025). Est. 0.005-0.015 BPB. Low complexity.
Higher-order complementary training — Record: 0.4416 BPB -- Complementary Training + Backoff N-gram Mixer #803 uses bigram statistics for loss reweighting. Using 4-gram or 7-gram statistics (matching the actual eval-time cache) would better align training with eval. Tokens easy for 7-grams but hard for bigrams currently get full training weight but will be handled by the cache at eval time. Est. 0.005-0.010 BPB. Low complexity.
Adaptive complementary alpha — instead of fixed COMPLEMENT_ALPHA=0.5, make
w = 1 - alpha * p_ngram. Tokens with p_ngram=0.99 get near-zero weight; tokens with p_ngram=0.1 get nearly full weight. Smooth weighting instead of hard threshold. Est. 0.003-0.008 BPB. Trivial complexity.GuidedQuant (gradient-aware PTQ). Integrates end-loss gradients into layer-wise quantization objectives, improving over standard GPTQ. Could replace the current GPTQ calibration step with gradient-informed block selection — relevant because GPTQ calibration quality directly affects the int5/int6 roundtrip loss. No competition submissions have tried this yet. Expected gain: 0.002–0.005 BPP over standard GPTQ (based on paper claims of consistent PTQ improvement). Source: OpenReview 2025.
Frequency-ordered tokenization. Reorder vocabulary by frequency and encode with variable-length integers before compression. Achieves 0.76–7.08 percentage point improvements on standard compressors (zlib, LZMA, zstd). Directly applicable as a post-hoc encoding step on the artifact — no training changes needed. Could save 200–500KB of artifact budget, enabling a larger model. Source: arXiv 2602.22958.
Larger model at lower bits (2-bit). Recent research confirms: a larger model quantized to 2-bit outperforms a smaller model at 4-bit given a fixed compression budget. Current competition meta is 11L/512d at int5/int6. A 14L/640d model at int3 could fit in 16MB and outperform — but requires int3 quantization infrastructure nobody has built yet. High risk, high reward.
Tier 2 — Top picks for pure neural track (sorted by expected value)
More Tier 2 ideas (lower EV or higher complexity)
x+f(x)with a connection matrix (n=2 → 16 params/layer, ~176 total). Richer than Catalytic Residuals or DenseFormer DWA. mHC adds Sinkhorn stability. Drop-in.x + β·proj(x) + f(x). Learned gate erases stale features before writing new ones. 3-5 ppl improvement at 124M. ~5.6K params for 11L. Addresses residual-path interference in quantized models.1-sqrt((t-T0)/(T+1-T0)). Outperforms linear, cosine, and other cooldown shapes in WSD schedules. Zero-cost swap.Tier 3 — Novel approaches, higher risk
Recently Discovered Techniques (Mar 28 research)
High relevance — directly applicable:
Medium relevance — higher implementation cost:
New Techniques (Mar 29 research)
Techniques from Recent Competition PRs (Mar 30)
(grad @ W_down) * act_gradinto GEMM epilogue via Epilogue Visitor Tree. Intermediate never touches HBM. Hopper-only. Complements forward Triton fusion.out = hidden + sigmoid(g) * skip. Lets model tune encoder-decoder blending per layer. ~5 params for 5 skip paths.Tokenizer Optimization (Validated by #1143 Scylla — 1.0806 BPP)
The biggest single-technique gain in the competition came from tokenizer choice, not architecture or quantization. #1143 achieved 1.0806 BPP (beating #1089's 1.1086 by 0.028) primarily through a TokenMonster-derived tokenizer.
Why it matters disproportionately at this scale: The embedding layer costs V*d parameters. At V=1024, d=384: ~295KB quantized. At V=32K: ~9.2MB — over half the 16MB budget. The parameter golf regime makes vocabulary size a first-order architectural decision.
TokenMonster vs BPE/Unigram: TokenMonster uses ungreedy multi-branch search (6 parallel branches scored per position). Produces ~37.5% fewer tokens at equivalent vocab size compared to BPE. BPE's greedy merges are known suboptimal (Bostrom & Durrett, EMNLP 2020).
Compression is necessary but not sufficient: Schmidt & Reddy (EMNLP 2024) showed maximum compression does not maximize model performance. The winning tokens are those aligned with what the model can learn, not those that minimize sequence length. #1143's autoresearch found a better vocabulary than pure compression optimization.
Open directions:
Techniques from Latest Competition Frontier (Mar 31)
(1-p)^2 * (-log p)down-weights easy tokens, focuses gradient on hard tokens. Rarely used in LLM training — conventional wisdom says it hurts calibration. But for BPB optimization, hard tokens dominate the loss.The Path Below 0.9 BPP (from #1184's 0.9485 baseline)
Best-case realistic estimate: Stacking SLOT + QK-Gain + TTT on #1184 → ~0.915-0.925 BPP. Sub-0.9 requires at least one breakthrough.
New technique: Over-Encoding (OE) — Keep Scylla's 998 output vocab but use hierarchical n-gram input embeddings (sum of 1-gram + 2-gram + 3-gram tables). Creates exponentially larger effective input vocab with <5% memory overhead. Log-linear loss reduction reported. Could be the key to sub-0.9.
SLOT scaling insight: More optimization steps (8→32) are nearly free since features are cached. Rank-4 delta (2048 params) and multi-layer delta (layers 9-11) are unexplored extensions with medium-high upside.
What Doesn't Work
Three failure patterns. (1) Throughput cost exceeds quality gain. In a 600s budget, anything adding >10% step overhead needs >10% per-step improvement to break even. QAT (#236: 115ms vs 67ms baseline), NorMuon (#236: 110ms), and MTP (#212, #236: 86ms) all fail this test. Partial reversal: #1031 uses MTP as auxiliary-only training signal (2-head, weight=0.1, discarded at export) with -0.0037 BPP claimed at zero artifact/eval cost — a different usage pattern (1 seed, unvalidated). (2) Mechanism redundancy. Stacking two techniques that extract the same signal yields diminishing returns — TTT+XSA underperforms XSA-alone (#290 vs #265), error-guided TTT doesn't improve over uniform TTT (#296), EMA without XSA hurts (#201). (3) Regime incompatibility. Techniques optimized for int6 break under different weight representations — the standard stack (XSA, SmearGate, WD, EMA/SWA, TTT) all fail on ternary (#367), and recurrence amplifies quantization error 900× (#363).
2:4 Structured Sparsity (#1105): +0.672 BPB. Definitively dead at competition scale.
Turbo-Muon on 8xH100 (#1105): +0.0018 BPP worse AND artifact over 16MB. Early convergence advantage at step 500 doesn't hold at 7000+ steps.
SLOT causality concern (#1105): @abaybektursun found shared delta optimized over all positions then applied to all positions leaks future tokens. Removed from their submission. However, #1176 and #1172 use SLOT successfully — implementation details may differ.
1xH100 not a viable proxy (#1186): 8x fewer optimizer steps means results don't transfer to 8xH100.
SGD+momentum TTT (#1186): +0.065 BPP vs AdamW. Use AdamW for TTT.
Scale deception (#1227): Local experiments can be 180° wrong at full scale. SSM hybrid showed -18% CE improvement at dim=192 but was +2.7% BPP worse at dim=512 on H100. Systematic bias, not noise. Takeaway: always validate on the target hardware and model size.
Product Quantization (#1227): +292% BPP. Catastrophic at competition scale.
PAQ Logistic Mixing (#1227): BPC=19. Fundamentally broken for multi-class prediction — compression-domain techniques don't transfer to neural LM output heads.
Complementary Training (#1227): +2.6% BPP at full scale. Does not help.
LN Scaling (#1227): +11.4% BPP. Harmful.
QAT as regularizer (#1227, positive): Quantized model beats float32 by 0.66%. QAT can improve BPP, not just preserve it — acts as beneficial regularization.
The Current Baseline Stack
The foundation that most competitive submissions share. Worth noting: several top submissions diverge from consensus in specific ways that paid off — #180 used int5 (former official SOTA), #236 used 524K batch instead of 786K, #76 dropped QAT and raised LR, #265 added XSA from a recent paper. The meta is a strong starting point, but the data shows room to improve individual components.
The core five: Integer quantization (int6-all or int5-MLP/int6-attn) + MLP 3x expansion + sliding window eval (stride=64) + zstd-22 compression + precision passthrough for sensitive layers (usually FP16 tied embedding; #236 uses int8 to fund MLP capacity). Near-universal across all competitive submissions, though quant precision varies — #76, #267, and former SOTA #180 use int5-MLP to fund larger BigramHash or extra layers.
Near-consensus optimizer settings: Muon momentum 0.99 (warmup from 0.92 over 1500 steps), halved LRs (matrix=0.02, scalar=0.02, embed=0.03), warmdown 3000 iters, grad clip 0.3. Most top submissions use these. Exceptions: @unixmadtoonslab's #76 (1.1468) uses higher LRs (0.03) and lower momentum (0.97). @saml212's #236 (1.1400) used 524K batch instead of 786K, gaining 0.017 BPB via more gradient updates. However, #375's systematic study on the #315 frontier base found 786K > 524K by 0.004 BPB (3-seed) — at the frontier, total tokens matter more than gradient steps. The optimal batch size is stack-dependent: 524K helps Tier 2-3 stacks; 786K helps XSA+EMA frontier stacks.
Part of the top stack: SmearGate + BigramHash + OrthoInit — used by most top validated entries. Requires OrthoInit to work (per #212's ablation). 11 layers + WD 0.04 + weight averaging (SWA or EMA). The standard-arch frontier (#414, 1.1228) builds on EMA + XSA4 + GPTQ-lite + Tight SWA + VE128 + Partial RoPE + LN Scale + Late QAT. The overall non-TTT frontier is now #609 (1.1154, XSA-all + Full GPTQ + Selective Pruning + Parallel Muon, @saml212).
Common but not universal: QAT with STE (~half), SWA (~17/49 validated), NorMuon (~3/49), FA3 (~13/49).
The Core Five Explained (for newcomers)
1. Int6 Quantization (instead of Int8)
Standard post-training quantization maps each weight to an 8-bit integer (256 levels). Int6 uses only 6 bits (64 levels, range [-32, 31]) with per-row scale factors, then compresses with zstd (level 22) instead of the baseline's zlib-9. Int6 frees ~25% more artifact space than int8, reinvested in a bigger model. Some submissions keep sensitive layers in fp16 (tied embedding) or int8 (embeddings) to limit compounding precision loss.
Origin: @nanlliu introduced int6 mixed precision in #39.
2. MLP 3x Expansion
The baseline uses 2x MLP expansion (hidden dim 1024 for 512-dim model). Top submissions use 3x (1536). Wider MLP = more expressive capacity, funded by int6 artifact savings.
Origin: @jfprincz in #70 (Mar 19 08:57 UTC). @saml212 independently reached the same insight in #61 later that day.
3. Sliding Window Evaluation
Overlapping windows (stride=64, window=2048) give each scored token 1984+ tokens of context vs minimal context with non-overlapping chunks. Purely eval-time. Worth 0.034 BPB per @samacqua's ablation in #77.
Origin: @mattqlf in #50. Stride debate: stride=256 gives marginally better BPB at 4x less eval time (#114). Doc isolation hurts at stride=64 — use flat-stream eval (#199).
4. FP16 Tied Embedding
The tied embedding matrix (input + output) is uniquely sensitive to quantization — errors compound in both directions. Keeping it in fp16 (~1MB) is the single highest-value precision decision.
Origin: @chonchiog in #42.
5. Zstd-22 Compression
Zstandard at level 22 squeezes int6 data significantly tighter than zlib-9 — enough to fit ~1-2M more parameters. Compression happens once after training; decompression is fast. Free lunch.
The Path Down: What Separates Each Tier
Post-enforcement (Mar 27), the competition has bifurcated into two tracks: n-gram cache + neural base (record submissions from 0.44 to 1.05 BPB, though many face compliance scrutiny) and pure neural frontier (1.05-1.12 BPB). Official SOTA is now 1.1147 (#1019). Novel eval-time methods like TARA (#1055, closed for causality violation) and DeltaNet Crawler (#1047, 0.8822, causality concerns flagged) explored new directions but have not yet produced compliant results.
Tier 1: Tweaking the Baseline (1.20–1.22 BPB)
Submissions in this range make one or two changes to the baseline: a longer sequence length, a learning rate sweep, a warmdown adjustment. The approach is "how do I improve this model?" — treating the baseline as mostly correct and looking for low-hanging fruit.
This works for the first 0.02 BPB, but hits a wall fast. The constraint isn't hyperparameters — it's the artifact budget. At int8+zlib, you can't fit enough model capacity to go further. Many submissions in this range are also on non-standard hardware (RTX 4090, Apple Silicon, 1xH100), which limits training tokens and disqualifies from the record track.
What to do if you're here: Adopt the core five (int6, MLP 3x, sliding window, FP16 embed, zstd-22) as a package. Each technique is well-documented in the deep dives below. Together they're worth ~0.05-0.07 BPB — the single biggest jump available.
Tier 2: Stacking Known Techniques (1.15–1.18 BPB)
These submissions adopted the core five and are assembling additional techniques: SmearGate, BigramHash, SWA, QAT, NorMuon. The approach is "what techniques exist and how do I combine them?" — surveying PRs, identifying high-impact components, and building a combined recipe.
This is effective: the leap from 1.22 to 1.16 is largely a stacking exercise. But submissions in this range often stop at "I added all the techniques" without investigating interactions. Common patterns: using SmearGate without OrthoInit (which hurts — per #212's ablation), running QAT from the start (which hurts — late QAT at 70-85% is better), or using SWA without sufficient weight decay (SWA shows no effect below WD=0.04).
What to do if you're here: Run ablations. Remove one technique at a time and measure the delta. You'll often find that one "improvement" is actually hurting because of interaction effects. Check your hyperparameters against the consensus (LR=0.02, momentum=0.99, warmdown=3000) but also against divergent successes like #76 (LR=0.03, momentum=0.97). Multi-seed validation (3 seeds) is essential — single-seed scores can be off by 0.002+ BPB.
Tier 3: Understanding Interactions (~1.120–1.15 BPB)
These submissions adopted the full technique stack and understood why each technique works. @jfprincz (#198 at 1.1326) is the canonical example: 11 layers + SmearGate + BigramHash + OrthoInit + WD 0.04 + SWA + FA3 assembled into a coherent system where each piece reinforces the others — WD makes weights compressible AND quantization-friendly, SmearGate+OrthoInit inject bigram context the small model can't learn from attention alone, and SWA smooths the weight landscape during warmdown.
The approach is "how do these techniques interact, and what's the optimal system?" Key markers of Tier 3 thinking:
What to do if you're here: Solidify your baseline with multi-seed validation. The primary path to Tier 4 is adopting XSA + Full GPTQ + EMA. The #609 stack (XSA-all + Full GPTQ + Selective Pruning + Parallel Muon + LeakyReLU²) reached 1.1154 but is non-record due to eval-time GPTQ — the techniques are valid if GPTQ calibration is moved into the 600s training budget. The official record SOTA target is #1019 at 1.1147. XSA + EMA is the shared infrastructure across all frontier submissions.
Tier 4: Pure Neural Frontier (<~1.120 BPB)
The official record SOTA is #1019 at 1.1147 (@abaybektursun). The #609 stack reached 1.1154 (non-record due to eval-time GPTQ) and GEPA #505 reached 1.1181 (artifact >16MB). Both demonstrate what's achievable with compliant implementations.
The key insight at Tier 4: EMA (0.997) outperforms standard SWA by 0.003 BPB (#375, 3-seed verified). #315 demonstrates that the XSA+EMA base still had headroom via careful regularization — Partial RoPE, LN Scale, and Late QAT each target a specific weakness.
What to do if you're here: Three options. (a) Beat #549 on pure neural: Adopt the #609 technique stack with GPTQ calibration inside 600s training budget. Remaining untried: Mousse optimizer, OptRot, systems opts (Liger-Kernel, 2:4 sparsity). (b) Add n-gram cache (→ Tier 5): The single biggest lever — 0.07-0.16 BPB from a legal backward-looking n-gram eval cache. (c) Legal TTT with compliant GPTQ: All frontier TTT submissions were closed for eval-time GPTQ. The recipe works if GPTQ calibration fits in 600s. GEPA + legal TTT at 1.0983 on 4xA100 (#628, 20k steps) — 8xH100 version untried.
Tier 5: N-gram Cache (Invalidated Mar 27)
The n-gram cache track was invalidated after discovery that hashed implementations scored only the correct token without full-vocabulary normalization. #978 proved correctly normalized n-gram achieves only 1.51 BPP (worse than neural baseline). 33+ PRs closed by @valerio-oai. Whether a correctly normalized eval-time statistical method can improve on pure neural remains an open question.
Technique Interactions Matter More Than Technique Count
A recurring pattern: techniques that work independently can fail in combination. TTT+XSA actively hurts (#303: +0.016 worse), EMA fails without XSA (#201) but succeeds with it (#287), and 12L fails at seq2048 but works at seq1024 (#219 vs #76). #474 confirms this extends to newer techniques: VRL + Gated Attention + Catalytic Residuals stacked on a 12L SWA base (no XSA, no EMA) yielded 1.1690 — worse than the same base without them (1.1466). Frontier techniques are optimized for the frontier base; applying them to weaker bases produces negative or null returns.
The untried combinations above should be evaluated against your specific model's weaknesses, not applied blindly. XSA + EMA appears to be a prerequisite for most newer techniques (VRL, GA). For the pure neural track, the strongest remaining candidates are systems optimizations (fused kernels, 2:4 sparsity — throughput gains with significance waived) and compression innovations (OptRot, entropy-coded weights). For the overall frontier, n-gram eval cache is by far the highest-impact lever available.
Val-Data & TTT Rulings (Mar 20-28)
Val data ruled out (Mar 20, @0hq): Val tokens cannot be in the artifact. Paid prefix (#168), error correction (#108), val-only training all banned for record track. Now in README FAQ.
TTT ruling (Mar 20, @0hq on #152): Only backward-looking TTT allowed — adapt on tokens already graded, not future tokens. Pre-eval adaptation invalid. Causal TTT (#267-style) remains allowed. In README FAQ.
Mar 22, @cocohearts on #317: TTT is "not in the spirit of the challenge." Broader organizer signal — even backward-looking TTT may face scrutiny.
Mar 23, @0hq on #402: Explicit TTT clarification — token-stream model is correct. You may use any preceding eval tokens already graded. You may NOT re-order the evaluation set. Invalid TTT PRs (train-on-val-then-measure) will be closed. Auto-review process being built.
Mar 23, @cocohearts: #374 rejected for insufficient statistical significance vs new SOTA. #505 needs packaging fixes.
Mar 24, @valerio-oai — enforcement sweep (15+ PRs closed). Two categories: (1) TTT information leakage: multi-epoch TTT with min-NLL selection, and adapting-then-scoring same tokens, both ruled equivalent to "training on the val set." #593, #576, #573, #568, #596, #605, #614, #620, #518, #548 closed. (2) Training data at eval time: GPTQ calibration using training data during eval budget disallowed. #593, #576, #569 closed for this. Calibration must count within training 600s. #589 ruled valid but closed — fails 0.005-nat threshold vs #549 SOTA. valerio-oai confirmed: "TTT is a valid approach in theory" but "very easy to unintentionally leak val data into."
Mar 25, @valerio-oai — second enforcement sweep (issue #677). Comprehensive audit. (1) Eval-time GPTQ: Training for full 600s then doing GPTQ calibration afterward (even 3-4s) is "accessing training data at eval time" — disallowed. #606, #615, #626, #639, #656 closed. (2) N-gram eval cache ruling: The concept is "directionally legal" — building a cache from already-scored tokens is allowed. The specific #659 implementation was illegal (hindsight selection: comparing n-gram vs LM on the true next token). Legal alternatives: fixed-weight blending or entropy-adaptive alpha (using model uncertainty, not ground truth). (3) #706 flagged: @valerio-oai told @newjordan that #706's GPTQ calibration still runs after 600s training time — needs fix. (4) Broad invalid TTT list: #410, #415, #417, #442, #462, #481, #486, #517, #518, #532, #555, #581, #595 all flagged for adapting on validation before the reported eval pass.
Mar 27, @valerio-oai — mass n-gram cache closure (33+ PRs, #677). Hashed n-gram caches disallowed: they score only the correct token via hashing without normalizing over the full token distribution, producing invalid probabilities. Two-pass rescoring (score → TTT → rescore) explicitly disallowed as "training on the eval set." PRs closed include #846, #853, #868, #869, #870, #876, #881, #888, #893, #900, #907, #912, #918, #982, and many more. Only reviewing PRs after #988 for potential merging. Recommended: @NoesisGenesis's formal criteria for valid causal prediction — (a) distribution depends only on artifact + strict prefix, (b) full normalized distribution over token vocabulary required before scoring, (c) score computed from pre-update probability only, (d) single left-to-right pass.
Mar 28, @valerio-oai on #728: Val-calibrated GPTQ "breaks autoregressivity" — disallowed. Self-generated calibration data (as in #1019) is "probably legal." PR left open pending fix.
Mar 28, @valerio-oai on #991: Closed for double-pass TTT (score, train, rescore same tokens).
Mar 28, @valerio-oai on #1028: GPTQ calibration running after 600s training cap = accessing training data at eval time. Author confirmed bug and resubmitted as #1047 with GPTQ within budget. Under organizer review.
Technique Deep Dives
The Muon Optimizer Family
Muon (MomentUm Orthogonalized by Newton-Schulz) is the optimizer at the heart of this competition's baseline, created by Keller Jordan for the NanoGPT speedrun. It runs standard SGD with Nesterov momentum, then post-processes each 2D parameter's gradient update by replacing it with the nearest orthogonal matrix via Newton-Schulz iteration. Intuitively: compute the gradient direction, then "clean it up" so the update is maximally informative without redundant directions. It's equivalent to steepest descent under the spectral norm, which improves the conditioning of the optimization landscape. ~35% faster training than AdamW on language models.
NorMuon extends Muon by adding per-neuron adaptive learning rates from accumulated second-order statistics. Vanilla Muon can produce updates with highly non-uniform norms across neurons, causing some neurons to dominate training. NorMuon normalizes row-wise after orthogonalization, combining Muon's conditioning benefits with Adam-style balanced per-neuron learning. It also improves distributed scaling by avoiding full momentum gathering across GPUs. Used by @mtybadger (#122), @vmfunc (#89), @abhishekgahlot2 (#137), and others.
Muon Weight Decay — The competition baseline's Muon optimizer has no weight decay. Decoupled weight decay for Muon (
p.mul_(1 - wd * lr)) existed in modded-nanogpt since Nov 2025, but wasn't in the baseline. @notapplica was the first to bring it into this competition in #60, improving BPB from 1.2160 to 1.2094. Weights stay smaller and better-distributed, improving both generalization and compressibility.Post-enforcement status (Mar 27): The Mar 27 enforcement sweep closed 33+ n-gram cache PRs after discovery of the normalization bug — implementations scored only the correct token without full-vocabulary normalization, producing artificially low BPP. #978 proved that properly normalized n-gram achieves only 1.51 BPB (worse than neural baseline). The question of whether a correctly normalized eval-time statistical method can improve on pure neural remains open.
Quantization-Aware Training (QAT) with STE
Instead of training in full precision and quantizing afterward, QAT simulates quantization during training. In the forward pass, weights are rounded to their quantized values. The problem: rounding is non-differentiable, so gradients can't flow through it.
The Straight-Through Estimator (STE) solves this by pretending the rounding operation is the identity function during the backward pass. It's mathematically "wrong" but works remarkably well — the model learns weight configurations that are robust to precision loss because it's been "seeing" quantized weights throughout training.
Late QAT outperforms full-training QAT: The later, the better. @trovatochris (#117) activates at 70%, @mohosy (#130) at 75%, @unixmadtoonslab (#76) at 85%. #76 even dropped QAT entirely at 12L (1.1468), finding WD=0.04 alone sufficient. @jfprincz's #315 pushes this to the extreme: STE activates only in the final 4% of training (lr_scale < 0.1, during low-LR warmdown). This cuts the int6 roundtrip gap to ~0.007 BPB while preserving full-precision convergence. The lesson: QAT activation is a spectrum — later = cleaner convergence, better int6 gap.
Int8 vs int6 QAT tradeoff: @mrdavtan's ablation in #145 shows that int8 QAT is not worth it under the 10-min wallclock cap. The
torch.quantilecall for exact percentile matching adds ~20% per-step overhead (64ms → 77ms), costing ~2,000 training steps. Result: 1.2052 BPB with QAT vs 1.1925 without — the lost training tokens hurt more than closing the ~0.007 int8 quantization gap. Int6 QAT, however, likely pays off because its larger ~0.01+ BPB gap justifies the overhead — confirmed by #128 and #137.SmearGate & Bigram Hash Embedding
@unnir introduced SmearGate in #102 and refined it in #135. This appears to be a novel technique for this competition — no published papers found.
SmearGate: A tiny learned gate (~512 params) that blends each token's embedding with the previous token's. This injects bigram (two-token) context directly into the embedding layer before the transformer starts processing. Normally a transformer must discover token pair relationships through self-attention; SmearGate provides this signal for free.
Bigram Hash: A hash table (commonly 2048-10240 buckets, dim=128, projected to 512) that maps token pairs to learned embeddings. Together with SmearGate, this gives the model token-pair awareness at nearly zero parameter cost.
@unnir's original combination with orthogonal initialization achieved 1.1539 BPB in #135. @jfprincz's #198 (1.1326) extended this with 11L + SWA + FA3 + WD 0.04, and #287 (1.1280) extended further with XSA + EMA.
OrthoInit appears critical for SmearGate. @mrdavtan's ablation in #212 found that adding SmearGate + BigramHash without OrthoInit hurt BPB (1.1739 vs 1.1708 without). Every successful SmearGate submission uses OrthoInit — the two techniques may be co-dependent.
Exclusive Self-Attention (XSA)
XSA (arXiv:2603.09078, Shuangfei Zhai, 2026) removes self-value bias from attention output via orthogonal projection. In standard attention, each token's value vector contributes to its own output — XSA subtracts this self-component, forcing the model to rely on information from other tokens. Applied to the last 3-4 layers only ("Partial XSA"), where self-attention bias is highest.
Zero parameters, minimal overhead. @unnir's #265 GQA-aware implementation reduces XSA overhead from ~7ms/step to ~2ms/step. Near-universal among frontier submissions. Best non-TTT (#609, 1.1154) uses XSA on all 11 layers; official SOTA (#1019, 1.1147) uses XSA-all.
XSA coverage depth: 4 layers appears near-optimal. @gowtham0992's #478 tested XSA on ALL 11 layers: 1.1268 (3-seed) vs XSA-4 at 1.1327 on the same base (−0.006 from XSA-all). But #414 (XSA-4 + VE128 + Partial RoPE + LN Scale) reaches 1.1228 — better than #478's XSA-all(11) at 1.1268. XSA-all adds ~3ms/step overhead (−230 steps), and removing self-value from ALL layers may degrade the model's own-representation capacity. The progression: 3 layers (#265: 1.1307) → 4 layers (#414: 1.1228) → 11 layers (#478: 1.1268) suggests 4-6 layers is the sweet spot for non-TTT. However, #609 (1.1154, best non-TTT) uses XSA-all(11) and #606 (1.1162, best legal TTT) also uses XSA-all — at the current frontier, XSA-all with Full GPTQ overcomes the overhead penalty.
Test-Time Training (TTT)
@samacqua introduced a creative approach in #77: adapting the model during evaluation.
For each validation document, rank-8 LoRA (Low-Rank Adaptation) adapters are trained on the document's own text using only backward-looking context (no data leakage). The model essentially "studies" each document briefly before being scored on it. LoRA makes this practical by only training tiny low-rank matrices (~1.5% of params) rather than the full model, enabling batched per-document adaptation within the eval time budget.
Original #77 ablation showed TTT itself adds ~0.003 BPB on early baselines (most gain came from doc isolation + sliding window). Full-model SGD TTT (#152) was ruled invalid by @0hq — only backward-looking (score-first) TTT is legal. The best legal TTT submissions (#606 at 1.1162, #615 at 1.1169) were later closed for eval-time GPTQ on training data (Mar 25 sweep).
TTT on XSA+EMA is a spectrum, not a binary. On SmearGate bases: #254 shows 0.014 BPB gain. Three XSA+EMA data points, sorted by base strength: (1) #317 (weak base, pre-quant 1.1581, no FA3): TTT gains 0.024 BPB. (2) #338 (@alertcat, #315 base — frontier at 1.1250, Partial RoPE + LN Scale + Late QAT): TTT neutral ±0.001 (3 seeds). (3) #303 (@sseanliu, #287 base — 1.1280, without #315's additional regularization): TTT +0.016 BPB worse. The pattern suggests TTT interacts with how tightly converged the base model is: under-trained bases benefit from local adaptation; over-regularized frontier bases are disrupted; the current frontier (#315) sits in a neutral zone. #338's neutral result is informative — it means TTT is not a meaningful lever at the frontier.
Reptile meta-TTT: gains on SmearGate, fails at frontier. @sseanliu's #296 shows 0.011 BPB on SmearGate models vs 0.001 naive. But #375 tested Reptile on #315's XSA+EMA base: +0.0076 worse, consuming 20% of training budget. The SmearGate gain does not transfer to the frontier. All three TTT variants (naive, MLP-only, Reptile) are now confirmed dead ends at ~1.125. Error-guided TTT is also negative — hardest tokens are genuinely unpredictable.
TTT optimizer recipe matters. @Christopher-Lee-McClendon's #461 (non-record, 4xA100) found that SGD+momentum(0.9), 3 epochs per 32K chunk, freezing first 2 blocks gets −0.0165 BPB TTT gain — 2.4× better than AdamW 1-epoch over all params (−0.0068 in their prior #456). Pre-TTT baselines nearly identical, so the entire improvement comes from the TTT recipe. This partially contradicts the #442 narrative (AdamW >> SGD) — the comparison is more nuanced: selective freezing + multi-epoch SGD with momentum can outperform single-epoch full-network AdamW.
Legal TTT survivors — none remain after Mar 25 sweep. #606 (1.1162) and #615 (1.1169) were closed — eval-time GPTQ calibration on training data. #576 (1.1164) closed in Mar 24 sweep. #573 (Multi-Pass min(NLL)) ruled invalid. All frontier TTT submissions used eval-time GPTQ and are now invalid. TTT optimizer matters for GPTQ: SGD TTT hurts Full GPTQ models (+0.030, #601), but AdamW with cosine LR works. The remaining legal TTT avenue requires GPTQ calibration within the 600s training budget.
Cosine TTT scheduling is a 3× multiplier. @mrdavtan's #481 (3-seed, 1.0970) introduced two TTT innovations on top of AdamW TTT: (1) cosine LR decay over 30 epochs — high LR early to repair quant damage, low LR late to refine; (2) per-layer LR groups based on measured quantization error — 3× base LR for MLP output projections (3.4× higher quant error), 0.5× for input projections. Result: TTT gain of −0.061 BPB vs #442's −0.019 with flat LR — a 3× improvement from scheduling alone. Pre-TTT ~1.158 (weaker base, FA2 not FA3). Also tested: focal loss and KL-divergence from pre-quant model — both failed to improve over CE.⚠️ Pre-eval TTT.
N-gram Eval Cache (the Mar 25 revolution)
The single biggest BPB lever discovered in the competition. During sliding window evaluation, a backward-looking n-gram cache is built from already-scored tokens and mixed with model predictions. The concept is simple: if the model has already scored "the cat sat on the", and the 5-gram "cat sat on the" was followed by "mat" last time, weight that prediction into the next token's distribution.
How it works:
p_final = (1-α) * p_model + α * p_ngramThree generations of implementation:
alpha = 0.05 + 0.55 * sigmoid(2 * (H - 4.0)). When the model is uncertain (high entropy), trust n-grams more. Additional −0.02 over fixed alpha. Combined with backoff: 1.1271 neural-only → 0.9674.Why it's so effective: Language has enormous local repetition — names, technical terms, formatting patterns — that a small transformer can't memorize but n-grams capture perfectly. The n-gram cache acts as a lossless "local memory" that costs zero artifact bytes (built on-the-fly from eval data).
Legality argument: All implementations claim score-first backward-looking compliance — the cache uses only previously-scored tokens, alpha depends on the model's own entropy (not ground truth), and there's no oracle selection. #702 cites @valerio-oai suggesting entropy-adaptive alpha as a legal alternative in the #659 review. But the technique hasn't been officially ruled on yet.
#738 adds kNN-LM: @gowtham0992 stores 512-dim hidden states in a GPU ring buffer and finds k=32 nearest neighbors for uncertain tokens. RBF kernel builds a non-parametric distribution. Additive −0.007 BPB on top of n-gram cache. Based on Khandelwal et al. 2019 (ICLR 2020). Captures semantic patterns that pure n-gram statistics miss.
Ablation data from #727:
#315's Techniques: Partial RoPE, LN Scale (Late QAT was inactive)
@jfprincz's #315 (1.1250) adds two effective zero-parameter techniques on top of #287's XSA+EMA base, gaining 0.0023 BPB. Note: Late QAT was also included in the code, but
torch.compileconstant-folded the_qat_enabledflag, making the STE branch dead code — Late QAT never activated (discovered by @152334H, confirmed in #453). Update: @wfproc (#1032) confirmed this dead-code bug persists in the current SOTA #549 codebase. A fix via tensor-scale STE actually worsened the int6 gap — suggesting WD+EMA already compensate for what QAT was supposed to do. The 0.0023 gain comes entirely from Partial RoPE + LN Scale.Partial RoPE (16 of 64 head dimensions). Rotary Position Embedding (RoPE) injects position information by rotating query/key vectors. Standard RoPE applies to all head dimensions. Partial RoPE applies to only 25% (16 of 64 dims) — the remaining 48 dims attend without position encoding. Why this helps: the position-free dims learn semantic similarity independent of token distance, improving generalization across different position ranges. The model can learn both "what things are" (position-free) and "where things are" (position-encoded) using different parts of the same head. Zero new parameters.
LN Scale (output scaled by 1/√(layer_idx+1)). After each RMSNorm, the output is multiplied by a layer-dependent scale factor that shrinks with depth. Layer 0: ×1.0; Layer 5: ×0.408; Layer 10: ×0.302. This damps the contribution of deeper layers to the residual stream, preventing later layers from "overwriting" early representations. Training is more stable — the model can use depth incrementally rather than being forced to route everything through deep layers. The 1/√(layer+1) schedule is related to the "depth scaling" used in some architecture papers. Zero new parameters.
Late QAT (STE enabled only when lr_scale < 0.1) —⚠️ was dead code in #315.
torch.compileconstant-folded the_qat_enabledclass attribute, so the STE branch never activated (discovered by @152334H, confirmed in #453). The concept is sound — late activation avoids corrupting Muon's momentum — but #315's actual gains came from Partial RoPE + LN Scale alone. Working Late QAT: @unnir (#374, scale<0.1), @signalrush (#414, threshold 0.15), @fbedev (#417). Downstream submissions copying #315's code may also have inactive Late QAT.The two active techniques (Partial RoPE + LN Scale) gain 0.0023 BPB vs #287 — statistically clear (3-seed variance 0.0005 BPB, t-stat -101.9 vs SOTA, p << 0.01).
Notable Non-Record Submissions
p += λ(p₀-p)) + Reptile (last 10%) + XSA3 + Pre-Q/K RMSNorm. TTT gain: -0.014 BPB (1.1660→1.1520). Adapts MLP only in last 3 blocks. Int5-MLP/int6-attn + BigramHash(10240). 1 seed.x + c*f(x), learned per-dim vector c). −0.024 BPB at zero overhead. 3-seed mean 1.1466. Built on #180.Idea Lineage & Diffusion (67 techniques tracked)
x + c*f(x), −0.024 BPB))alpha = 0.05 + 0.35 * sigmoid(2*(H-4))))Predictions & Commentary
SLOT is now mainstream. Nine record submissions use SLOT variants. Context-Only SLOT (#1217) proved causal with only −0.0002 BPP cost. Scored-Position SLOT (#1229) pushed to 0.9300 BPP with per-sample deltas. Expect SLOT in every competitive submission going forward.
Scylla + SLOT remains the obvious untried combination. #1184 (0.9485, Scylla alone) and #1229 (0.9300, SLOT alone) are the two best pending submissions. Combining them could push toward 0.92–0.93. Someone will try this soon.
Two separate frontiers have emerged. Standard-tokenizer submissions top out around 1.0979 (#1218, simplification approach). Custom tokenizers (Scylla) dominate overall at 0.9485. These are effectively different competitions now.
The official leaderboard is about to be disrupted. The gap between official SOTA (1.1147) and top pending (#1229, 0.9300) is 0.185 BPP. When these merge, most current "competitive" submissions will fall below the new 0.005-nat threshold. Expect a wave of invalidations.
Fused kernels help but aren't the moat. CUTLASS fusion gives ~2–3% step speedup, but custom tokenizers and eval-time adaptation (SLOT, TTT) dominate the gains. Systems optimization is necessary but not sufficient.
Simplification won the standard-tokenizer race. #1218 proved that removing TTT/QAT/hash embeddings and adding 4096-vocab + higher WD (0.085) beats complex stacks. The RMS-compression insight (R²≈0.99 between weight RMS and compressed size) is a genuine contribution — expect wider adoption of WD tuning for artifact size.
Negative results are accelerating. Focal loss, self-gen GPTQ, random MLPs, oscillatory recurrence all failed this cycle. The frontier is mature enough that most "clever ideas" lose to more training steps. Time-budget tradeoffs dominate: any technique that costs >5% of training time must deliver >0.01 BPP to be net positive.
Full Official Leaderboard (19 entries)
Validated against the SOTA at submission time.
All Record-Eligible Submissions (30 entries)
Sorted by BPB ascending.
15.5MB, 27M params. No TTT. Std0.0002.All Not Yet Validated Submissions (100 entries)
Competitive submissions that haven't demonstrated statistical significance.
Glossary
Changelog