Record: 11L NonTTT VR+GA MixedInt5/6: val_bpb=1.1428 (3-seed, 8xH100)

[Asukabot0]

Summary

Non-TTT submission with Value Residual (ResFormer) + Gated Attention + Mixed Int5/Int6 quantization.

3-seed mean val_bpb: 1.1428 (sliding window, stride=64, int6 roundtrip)

Approach

Built on PR #315 base (EMA + Partial RoPE + LN Scale) with two architectural additions:

• Value Residual (ResFormer): Cache layer-0 V vectors and mix into subsequent layers via learnable λ. 22 params per block, zero compute overhead. (arXiv:2410.17897, PR Non-record: Value Residual (-0.015 BPB) + Gated Attention (-0.003 BPB) with ablations #413)
• Gated Attention: Per-head sigmoid gate on attention output, suppresses attention sinks. ~37K params total. (arXiv:2505.06708, PR Non-record: Value Residual (-0.015 BPB) + Gated Attention (-0.003 BPB) with ablations #413)
• Mixed Int5/Int6 quantization: MLP middle layers use int5 [-16,15] for better zstd compression, edge layers + attention keep int6 [-32,31]

Key config

11L × 512 × 3xMLP, XSA4, EMA(0.997), Partial RoPE(16/64), LN Scale
Muon(momentum=0.99, lr=0.025, wd=0.04), warmdown=3000
TTT=OFF, Late QAT=OFF
Mixed int5/int6 + zstd-22

Results

Seed	Steps	ms/step	val_bpb	Artifact Bytes	<16MB?
1337	4622	129.82	1.14280	16,026,184	NO
42	4623	129.79	1.14281	16,339,774	NO
2025	4632	129.55	1.14287	16,244,044	NO
Mean	4626	129.72	1.14283	16,203,334	NO

Known Issues & Next Steps

This submission has a DDP performance regression we've identified and are fixing:

1. find_unused_parameters=True in DDP causes +53% step time (130ms vs expected ~85ms), reducing steps from ~7000 to ~4600. VR and GA params all receive gradients — this flag is unnecessary. Fix: set find_unused_parameters=False.
2. Artifact slightly exceeds 16MB (by 26–340KB). Fix: more aggressive int5 layer selection or magnitude pruning.

With these fixes we expect:

• Step speed: ~85ms → ~7000 steps in 600s
• val_bpb: ~1.12 range (matching single-GPU extrapolation)
• Artifact: <16MB

Will submit an updated run after fixes.

Hardware

• 8× H100 SXM 80GB
• 600s wallclock (all seeds completed within budget)
• Peak memory: ~27 GiB per GPU