Fuzzing 110 configurations

[bigwolfeman]

I started with an initial random search of 30 individuals then used that to inform an 80 configuration Bayesian fuzzing search of hyper parameters. I found Muon momentum to be the dominant factor in what changes loss values.

---

Optuna Causal Analysis:

Parameter Importance

• muon_momentum 82.2% ← DOMINANT FACTOR
• schedule_type 7.1%
• weight_decay 6.1%
• adamw_lr 2.1%
• grad_clip 1.2%
• muon_lr 1.2%
• warmup_ratio 0.5%

Winning Configuration (Trial 72, val_loss=4.6744)

Hyperparameters:

• muon_lr=0.0272 # Mid-range (was 0.024 default)
• adamw_lr=0.0118 # 2x higher than default (0.006)
• muon_momentum=0.862 # CRITICAL: lower than default (0.95)
• weight_decay=0.120 # Lower than default (0.2)
• warmup_ratio=0.059 # Added warmup (was 0.0)
• grad_clip=0.79 # Lower than default (1.0)
• schedule='cosine' # Changed from constant