[10min_16mb] 0.9641 BPB: LeakyReLU² + Score-First TTT + N-gram Backoff Cache

[skoustav35]

Submitting a new entry for the 10-minute 16MB track that achieves a 3-seed exact mean of 0.9641 BPB (1.6274 nats).

This improves upon the current merged 1.1147 BPB baseline (PR #1019) by 0.1506 BPB (0.2548 nats), which exceeds the required 0.005 nats threshold by ~51× (Welch t = -328.3, p ≪ 0.01).

Techniques Used

• Architecture: 11 Layers, 512 dim, GQA = 8H/4KV, MLP 3x, LeakyReLU(0.5)², XSA-5 (layers 6-10), Tied embeddings, Value Residual, Gated Attention, VE(128) on layers 8/9/10, MTP-2, BigramHash 2048.
• Eval-time N-gram Backoff Cache:
  • Multi-order backoff (orders 2–9), picking the highest matching order.
  • Laplace (add-1) smoothing: Ensures the returned probability is a proper normalized distribution over the vocabulary and does not depend on target-oracle knowledge.
  • Entropy-adaptive alpha scaling.
• Test-Time Training (Legal, Score-First):
  • SGD, 3 epochs, 32K token chunks, stride 64.
  • Tokens are scored strictly backward-lookingly before updates.
• Optimization & Quantization:
  • Muon + Adam split.
  • Int6 per-row quantization with LZMA compression. Late-stage CROWN-Q penalty.

Compliance & Margins

• Training Time: Seeds complete in 599,384, 599,761, and 599,618 ms (Note: logged train_time excludes initial compilation and 20 warmup steps).
• Artifact Size: 15,989,583 bytes max across seeds (well under 16,000,000 B).
• N-Gram Cache Legality: We note that this technique builds on the cache method seen in closed PR Record: First Legal Sub-1.0 BPB — Multi-order N-gram Backoff + Entropy-Adaptive Alpha (val_bpb=0.9674, 3-seed) #727, and explicitly acknowledge the ongoing discussion in issue Illegal submissions megathread #677 regarding eval-time caching methods. This implementation uses zero artifact bytes and is strictly backward-looking.

Reproducibility

The script resolves data paths relative to the repo root automatically.

SEED=1337 RUN_ID=seed_1337 VOCAB_SIZE=1024 \
torchrun --standalone --nproc_per_node=8 \
  records/track_10min_16mb/2026-03-31_LeakyReLU2_LegalTTT_NGramCache_XSA/train_gpt.py