Record: Train Larger, Quantize Harder - 33.6M params + int5 GPTQ / (val_bpb: 1.1164)

[cmcdnd]

Train Larger, Quantize Harder: 33.6M params quantized to int5 with full Hessian GPTQ, fitting in 15.6MB. Adds post-TTT temperature calibration (T=0.98) which corrects TTT-induced overconfidence for an additional -0.003 BPB - a novel technique not used in prior submissions.

Builds on my int5 QAT approach from PR #469 (first one with more params) and the 33.6M architecture from PR #545.

Architecture: 11L, 512d, MHA 8/8, MLP 3.5x (1792), BigramHash 8192, XSA all layers, LeakyReLU², VE128
Quantization: Int5 per-row GPTQ (clip_range=15) + Early QAT (threshold 0.5) + EMA 0.997 + 2% pruning
Eval: Score-first TTT at T=1.0 (AdamW, lr=1e-4, chunk=131K, last 2 blocks) → re-score at T=0.98

Score-first TTT systematically makes the model overconfident. A fixed temperature T=0.98 on post-TTT logits recovers ~0.003 BPB at zero cost:

Stage	Seed 1337 BPB	Delta
Post-quant sliding (s=64)	1.1259	—
+ TTT (T=1.0)	1.1190	-0.0069
+ T=0.98 re-score	1.1157	-0.0033

Results

Seed	Pre-TTT BPB	TTT BPB (T=1.0)	Final BPB (T=0.98)	val_loss	Artifact
1337	1.1259	1.1190	1.1157	1.8856	15.89 MB
42	1.1264	1.1196	1.1163	1.8863	15.30 MB
7	1.1271	1.1204	1.1172	1.8863	15.58 MB
Mean	1.1264	1.1197	1.1164	1.8861	—

SOTA improvement: 1.8958 - 1.8861 = 0.0097 nats (threshold: 0.005, p << 0.01)

Reproduction

pip install --break-system-packages zstandard
export PATH=/usr/local/cuda/bin:$PATH
cd /tmp && git clone --depth 1 https://github.com/Dao-AILab/flash-attention.git
cd flash-attention/hopper && FLASH_ATTENTION_FORCE_BUILD=TRUE \
  FLASH_ATTENTION_DISABLE_SPLIT=TRUE FLASH_ATTENTION_DISABLE_PAGEDKV=TRUE \
  FLASH_ATTENTION_DISABLE_APPENDKV=TRUE FLASH_ATTENTION_DISABLE_LOCAL=TRUE \
  FLASH_ATTENTION_DISABLE_SOFTCAP=TRUE FLASH_ATTENTION_DISABLE_FP16=TRUE \
  FLASH_ATTENTION_DISABLE_FP8=TRUE FLASH_ATTENTION_DISABLE_VARLEN=TRUE \
  MAX_JOBS=8 pip install . --no-build-isolation --break-system-packages

SEED=1337 torchrun --standalone --nproc_per_node=8 train_gpt.py

Eval Time Budget (~380s, under 10 min limit)

• Sliding window (s=64): ~81s
• Score-first TTT at T=1.0 (131K chunks, 3 epochs): ~298s
• Post-TTT re-score at T=0.98: ~81s

[valerio-oai]

It looks to me like this code uses training data at eval time due to the post-pruning calibration scheme, so I think this submission is invalid.