Record: Seed-Regenerated Random Model + Incremental N-gram Cache — val_bpb 0.0905

[vimeto]

Record: Seed-Regenerated Random Model + Incremental N-gram Cache — val_bpb 0.0905

val_bpb = 0.0905 (1 seed, additional seeds pending H100 access) | 15.09 MB | 8xH100 SXM

Results (8xH100 80GB SXM, PyTorch 2.7.1)

Seed	step_avg	steps	neural_bpb	blended_bpb	Artifact
1337	60ms	9,912	1.503	0.0905	15,093,968
42	pending
2025	pending

Additional seeds pending H100 access.

Key Innovation: Zero-Cost Base Weights

ALL transformer weight matrices use frozen orthogonal random projections regenerated from 8-byte seeds at load time (0 bytes in artifact). Only rank-64 LoRA adapters are stored (3.9 MB). The remaining 11 MB holds an incrementally-built INT16 n-gram cache (orders 2-7, 31B counts, 8-GPU all-reduce synced).

Why orthogonal: Prior work (PR #874) used Gaussian random bases but could not train past 5 layers. Our QR-decomposed orthogonal init preserves singular values = 1.0, enabling stable deep training.

Adapter quantization: Simple INT8 per-row gives quant gap of only +0.003 BPB (vs +0.006 for baseline INT6 GPTQ).

Incremental N-gram Cache (Zero Overhead)

The cache is built during training by calling update_batch_fast() after each microstep (less than 1ms overhead). After training, counts are all-reduced across 8 GPUs and LZMA-compressed into the artifact. At eval, the cache is frozen, no TTT.

We tested pre-filling from training shards at startup: 10x worse (0.996 BPB) due to pre-fill consuming 24-33% of the training budget.

Architecture

5L 512d, 8H/4KV, MLP 3.0, LeakyReLU(0.5) squared, rank-64 LoRA adapters, tied embeddings, vocab 1024

Credits

• PR Non-record: Random Linear Maps + Learned Adapters (val_bpb=1.93, 2.4MB artifact) #874 (@fielding) — Random linear maps concept
• PR Record: 0.0498 bpb - Packed Training N-gram Artifact + Learned Weighting Gate (updated)  #931 (@AnirudhRahul) — Packed n-gram artifact
• PR Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean) #549 (@abaybektursun) — LeakyReLU squared activation