Qwen3.5 MoE: MLP experts silently skipped during abliteration (attention-only, no warning)

[rocker-zhang]

Running Heretic v1.3.0 on a Qwen3.5-122B-A10B model, the abliterable-components summary lists only attn.o_proj (48 modules); the MoE expert MLPs are never ablated, with no indication that they were skipped.

The cause is in get_layer_modules (src/heretic/model.py:383-385): it iterates layer.mlp.experts, but in transformers 5.6 the Qwen3.5 MoE experts are a single fused nn.Parameter (Qwen3_5MoeExperts.down_proj), not an iterable of per-expert modules. for expert in layer.mlp.experts raises TypeError, which suppress(Exception) swallows, so zero MLP modules are matched and the run proceeds attention-only.

I saw from #206/#207 that #187 gives basic (attention-only) Qwen3.5 support and that fused-expert ablation should avoid direct weight modification, with ARA (#211) in progress. Since #202 notes MLP ablation is sometimes needed to bring refusals down, a user on these models currently gets no signal that the MLP path was skipped.

Would a diagnostic-only warning be welcome in the meantime (no weight modification): when a layer exposes an experts container but zero MLP modules are matched, print a one-time "ablation will be attention-only" warning? Or will ARA make this moot soon enough that it is not worth adding? I can send a PR for the warning if useful.

[p-e-w]

Indeed. This isn't a bug or an oversight, but a side effect of how our abliteration engine works.

Heretic performs abliteration by attaching LoRA adapters to modules via PEFT. Unfortunately, nn.Parameter is not a subclass of nn.Module – it's a subclass of torch.Tensor, and thus lacks the input/output hooks that Module provides and that PEFT relies on to modify the residual stream on-the-fly, without touching the model weights.

While the current ARA branch (#211) uses direct weight modification, we are actually going to move ARA to use LoRAs as well before merging it (see #332). So ARA will have this problem as well, but LoRAs just add too much value to not use them, while keeping full-weight modification as an option would add too much complexity.

My personal philosophy is that warnings should be informative and (ideally) actionable. Warning about Heretic not being able to target the MLP fails both tests, IMO. Such a warning would mean nothing to most users who are unfamiliar with transformer internals, and even if they did understand it, there would be nothing they can do about it. Whether this is even detrimental (and thus warn-worthy) at all also depends on both the model and the ablation method. Overall, it's hard to see what concrete value such a warning would add, especially since we already show which components we are targeting in the output.

Note that there are also other components that we might target in the future (such as convolutions in certain hybrid architectures), so it's not like certain components are "required" in a meaningful sense. We just modify what we are able to, and then use that to drive the optimization process.

[rocker-zhang]

That makes sense, thanks for the detailed write-up. The note that ARA will move to LoRAs too (#332) is the key takeaway: the fused nn.Parameter case stays unablated by design, not something to paper over with a warning. Agreed it would fail the informative-and-actionable bar, and the targeted-components output already shows what's covered. I'll close this.