Record: 11L LeakyReLU² + VRL + lzma — val_bpb 1.1229 (3-seed mean)#657

Closed

anthony-maio wants to merge 20 commits intoopenai:mainfrom

anthony-maio:submission/reproduce-414

Commits on Mar 24, 2026

Swap ReLU² → LeakyReLU(0.5)² in MLP activation

anthony-maio
and
claude
committed
Add Value Residual Learning (VRL) from ResFormer paper

anthony-maio
and
claude
committed
Minify code: 80KB → 60KB (remove TTT, comments, docstrings, whitespace)

anthony-maio
and
claude
committed
MLP 3.0x → 2.875x: hidden 1536 → 1472 (tensor-core aligned)

anthony-maio
and
claude
committed
Shrink artifact: bigram 2048→1536, control tensors fp32→fp16

anthony-maio
and
claude
committed
Switch to lzma compression, restore MLP 3.0x + bigram 2048

anthony-maio
and
claude
committed
Record: 11L LeakyReLU² + VRL + lzma, val_bpb=1.1234

anthony-maio
and
claude
committed
Update: 3-seed results — mean 1.1229, std 0.0005, all valid

anthony-maio
and
claude
committed

Commits on Mar 25, 2026