Canary AED adapter (encoder + transf_decoder LinearAdapter) drops words / fails to terminate sentence under 2.8.0rc0 — works correctly under 2.4.0 with identical weights

[kaniosm]

Describe the bug

A LinearAdapter (from nemo.collections.common.parts.adapter_modules) trained with NeMo 2.4.0 on canary-1b-v2, attached at both encoder.layers[*] and transf_decoder._decoder.layers[*] simultaneously (a single adapter, two module targets via the enc/dec config keys), produces materially degraded transcriptions when loaded into NeMo 2.8.0rc0 and run against the identical frozen base checkpoint — despite:

• The base model's full state_dict being byte-for-byte identical between the two environments (verified tensor-by-tensor across all 1670 parameters).
• The adapter .pt checkpoint (weights + __cfg__) being loaded successfully via AdapterModelPTMixin.load_adapters() with strict=True (no shape/key errors — the weights genuinely are the ones that were saved).
• The adapter's config being fully explicit (no NeMo-version-dependent defaults in play): norm_position: pre, dropout: 0.0, activation: swish, adapter_strategy: ResidualAddAdapterStrategy with stochastic_depth: 0.0, l2_lambda: 0.0.

Degradation shows up as dropped words and missing/incorrect sentence-terminating punctuation — consistent with destabilized autoregressive decoding rather than an acoustic/encoder issue. Enabling only the encoder-side or only the decoder-side half of the adapter individually does not reproduce this cleanly (both show some imperfection, on both NeMo versions, since the adapter was trained with both halves active together) — it's specifically the combination of both halves active simultaneously where 2.4.0 stays coherent and 2.8.0rc0 degrades sharply.

Steps/Code to reproduce bug

Minimal standalone repro (no training-service code involved) — run this once with nemo_toolkit==2.4.0 and once with 2.8.0rc0 installed, same Python env pattern, same base checkpoint and adapter file, same audio:

import torch
import nemo.collections.asr as nemo_asr

BASE = "<path-to>/canary-1b-v2.nemo"
ADAPTER = "<path-to>/adapter.pt"   # LinearAdapter targeting encoder ('enc') + transf_decoder ('dec')
AUDIO = "<path-to>/audio.wav"       # 16kHz mono, ~5s Greek speech

model = nemo_asr.models.ASRModel.restore_from(restore_path=BASE, map_location=torch.device("cpu"))
model.replace_adapter_compatible_modules()
model.eval()
model.freeze()

model.load_adapters(filepath=ADAPTER, map_location="cpu")
model.set_enabled_adapters(enabled=False)

def transcribe(label):
    out = model.transcribe(AUDIO, source_lang="el", target_lang="el", timestamps=False, taskname="asr", pnc="yes", batch_size=1)
    print(f"=== {label} ===\n{out[0].text}\n")

transcribe("base (no adapter)")

model.set_enabled_adapters(name="enc", enabled=True)
model.set_enabled_adapters(name="dec", enabled=True)
transcribe("enc+dec adapter (full)")

Note: with PyTorch 2.6+, TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=true must be set in the environment for load_adapters()'s internal torch.load() call to succeed (unrelated to this bug, just needed to reproduce).

Actual results observed (same base checkpoint, same adapter file, same audio, only NeMo version differs):

• NeMo 2.4.0, full adapter: "Ε Μάριος Ιακώβου είμαι. Θα ήθελα να ρωτήσω κάτι τη γιατρό Σολωμού." (complete, correctly terminated)
• NeMo 2.8.0rc0, full adapter: "Ε Μάριος Ιακώβου ήθελα να ρωτήσω κάτι γιατρό Σολωμού" (drops "είμαι. Θα" and "τη", no terminating punctuation)
• Base model (no adapter) on both versions: near-identical modulo trivial dither-induced jitter — confirms the frozen backbone itself is not the source of the divergence.

Expected behavior

A .nemo base checkpoint + adapter checkpoint that produces a given transcription under NeMo 2.4.0 should produce an equivalent-quality transcription under 2.8.0rc0, given that the underlying model weights are provably identical and the adapter config is fully explicit. At minimum, this combination shouldn't silently regress without any error/warning — the only anomaly during loading was No adapter compatible with the current module. Skipping adapter forward pass. logged once on the 2.4.0 run (worth investigating whether this indicates an already-existing partial mismatch that 2.8 handles differently).

Environment overview

• Environment location: Bare-metal (Linux, WSL2)
• Method of NeMo install: pip / uv pip, nemo_toolkit[asr]==2.4.0 for the 2.4.0 environment; the 2.8.0rc0 environment was installed from source (vendored NeMo checkout).
• Reproduced on CPU (map_location="cpu") in both environments to eliminate GPU/kernel nondeterminism as a factor.

Environment details

• OS: Linux 5.15 (WSL2)
• Python: 3.12
• PyTorch: 2.9.0.dev (nightly, cu129) in the 2.8.0rc0 env; 2.13.0 in the 2.4.0 repro env (only used to isolate NeMo version — base model weight identity and the divergence itself were separately confirmed against the project's actual pinned torch build)
• Base model: canary-1b-v2 (Canary AED / EncDecMultiTaskModel)
• Adapter: LinearAdapter (in_features=1024, dim=128, norm_position=pre) attached to both encoder.layers[*] and transf_decoder._decoder.layers[*]

Additional context

A second adapter for the same base model, trained natively under 2.8.0rc0 with a different architecture (RelPositionMultiHeadAttentionAdapter on the encoder + LinearAdapter on the decoder), does not show this degradation under 2.8.0rc0 — consistent with it having been calibrated against whatever 2.8's decoding path actually does, rather than this being a general "adapters are broken" issue. This suggests the regression is specific to how a ResidualAddAdapterStrategy-based LinearAdapter simultaneously active on both the Conformer encoder and the AED transformer decoder interacts with the autoregressive generation path (KV-caching / context_len_for_AR_decoding) between these two NeMo versions, rather than a generic adapter-loading problem.