ESMC-6B main loads as uninitialized weights via the transformers ESMC classes #405

biohub/ESMC-6B was recently republished and since then, loading it through
the transformers ESMC classes silently produces a model whose entire trunk is
unpopulated with random values. Nothing raises, so the model loads with warning
and then produces garbage embeddings.

This looks like an unintended consequence of #333. That fix appears to
have fixed the SDK path and broken the transformers path. IIRC the two loaders want
different key layouts and the repository can currently satisfy only one.

from_pretrained reports the mismatch as a warning and continues:

Some weights of ESMCForMaskedLM were not initialized from the model checkpoint
at biohub/ESMC-6B and are newly initialized: ['esmc.embed.weight',
 'esmc.transformer.blocks.0.attn.k_ln.weight',
 'esmc.transformer.blocks.0.attn.layernorm_qkv.layer_norm_weight',
 'esmc.transformer.blocks.0.attn.out_proj.weight',
 'esmc.transformer.blocks.0.attn.q_ln.weight',
 'esmc.transformer.blocks.0.ffn.fc1_weight',
 'esmc.transformer.blocks.0.ffn.fc2_weight', ...]

This random initialization even introduces some NaN parameters, which I used to diagnose
which cached snapshots are legit and which aren't:

Because ESMFold2Design reuses one ESMC trunk across every inversion and critic
model, a single bad trunk poisons all of them.

Downstream symptom

Those NaN weights are what led to a downstream crash when running ESMFold2:

File "transformers/models/esmfold2/modeling_esmfold2_common.py", line 1754, in _weighted_rigid_align
    U, _, Vh = torch.linalg.svd(H32, driver="gesvd" if H32.is_cuda else None)
torch._C._LinAlgError: linalg.svd: (Batch element 0): The algorithm failed to
converge because the input matrix is ill-conditioned or has too many repeated
singular values (error code: 3).

H32 there is (1, 3, 3) and every element is NaN. The error names SVD
convergence, which sends you looking at cuSOLVER, drivers and GPU models rather
than at weight loading. Based in this I think that a plain inference user gets no error at all,
just the warning and then garbage embeddings.

Environment

• esm 3.3.0 (ba4d712)
• transformers fork at ef32577f55da (current main of Biohub/transformers)
• torch 2.8.0+cu128, H100 and A100-80GB (identical on both)

Reproduction

from transformers.models.esmc.modeling_esmc import ESMCForMaskedLM
import torch

model = ESMCForMaskedLM.from_pretrained("biohub/ESMC-6B")  # warns, does not raise
bad = {n: int(torch.isnan(p).sum()) for n, p in model.named_parameters()
       if torch.isnan(p).any()}
print(bad)   # non-empty on 7d0f99a8; empty on 45b0fa5d

Pinning revision="45b0fa5d7fb06faefbd5e3b89bdcef35d564e79a" restores correct
behaviour.