RecurLoRA: Quantization-Stable Shallow Recurrence with Low-Rank Corrective Adapters

[Tanush1912]

Summary

• RecurLoRA: layers 4–5 repeated once (11 -> 13 virtual layers) with rank-2 LoRA corrections on attention projections + RMSNorm + learnable alpha
• Built on PR Record: Split-LR + BigramHash(2816x160) + Full GPTQ + Brotli — val_bpb 1.1105 (3-seed mean) #1179 stack with validated hyperparameter improvements (LeakyReLU 0.9, QK-Gain 4.0, score-first TTT, warmdown 4000)
• LoRA overhead: 28KB (0.18% of 16MB budget), kept as fp16 passthrough, avoiding additional quantization error

Why this direction

Weight sharing has consistently failed in this competition due to quantization error accumulation across repeated layers (e.g. PR #363: +4.3 BPB at 3 cycles).

However, PR #686 demonstrated that shallow recurrence (<=2 repeats) remains stable under int6 quantization (~1.1182 BPB), suggesting that limited reuse is viable.

RecurLoRA builds on this by introducing per-pass low-rank corrective adapters:

• Shared base weights capture global structure
• Rank-2 LoRA adapters specialize each pass (attention only)
• RMSNorm + learnable alpha mitigate residual drift

This enables increased effective depth (11 -> 13 layers) without incurring the instability of deep recurrence, effectively reallocating parameters from duplicated layers into increased depth under a fixed 16MB budget.

Status

Implementation complete and validated for:

• Forward/backward correctness
• Gradient flow across recurrent passes (warm-initialized LoRA: A and B active from step 1)
• Parameter budget compliance (28KB overhead)

Full training runs (3 seeds + ablations) queued pending compute.

Test plan

• Baseline: hyperparameters only (no recurrence)
• Recurrence only: confirm parity with PR Record: Depth Recurrence (layers 4 and 5 repeated): val_bpb 1.1182 #686 (~1.1182 BPB)
• RecurLoRA: recurrence + rank-2 LoRA (3 seeds)
• Alpha sweep: [0.3, 0.5, 0.6, 0.8]
• Report mean +/- std across seeds