canary-1b-v2 systematically drops word-final Greek sigma «ς» (U+03C2) — the final-sigma character is absent from the SentencePiece tokenizer vocabulary

[nickprotop]

Describe the bug

When transcribing Greek (source_lang="el", target_lang="el"), nvidia/canary-1b-v2 produces
otherwise-accurate Greek but omits the word-final sigma «ς» (U+03C2) on every word. Medial
sigma «σ» (U+03C3) is transcribed correctly — only the word-final form is affected, and it is
dropped entirely (not converted to «σ»).

Examples (reference → hypothesis):

Reference	Hypothesis
εκτός	εκτό
ρεύματος	ρεύματο
ασθενής	ασθενή
οργανισμός	οργανισμό
αγγλικές	αγγλικέ

Over a 101-word sample, 0 output words ended in «ς», while the references contained many.
This inflates naive WER dramatically (≈24% on a FLEURS-Greek subset) even though the underlying
transcription is otherwise correct — a sigma-insensitive WER of the same output is ≈7.8%.

Root cause (verified)

The unified SentencePiece tokenizer shipped inside canary-1b-v2.nemo has no token for the
final-sigma character «ς» (U+03C2). It is therefore unrepresentable in the model's output
alphabet: at encode time «ς» maps to an out-of-vocabulary piece, and at decode it becomes the
⁇ unknown marker, which is stripped from the final text — so the character is lost.

Reproduced by extracting the tokenizer from the .nemo archive and inspecting it directly:

import glob, tarfile, tempfile, os, sentencepiece as spm
arc = glob.glob(os.path.expanduser(
    "~/.cache/huggingface/hub/models--nvidia--canary-1b-v2/snapshots/*/canary-1b-v2.nemo"))[0]
d = tempfile.mkdtemp()
with tarfile.open(arc) as t:
    m = [n for n in t.getnames() if n.endswith("_tokenizer.model")][0]
    t.extract(m, d)
sp = spm.SentencePieceProcessor(model_file=glob.glob(d + "/*_tokenizer.model")[0])
pieces = [sp.id_to_piece(i) for i in range(sp.get_piece_size())]

print("vocab_size:", sp.get_piece_size())
print("has final sigma ς (U+03C2):", any("ς" in p for p in pieces))   # False
print("has medial sigma σ (U+03C3):", any("σ" in p for p in pieces))  # True
print(sp.encode("εκτός", out_type=str))          # ['▁εκ', 'τό', 'ς']
print(repr(sp.decode(sp.encode("εκτός"))))       # 'εκτό ⁇ '
print(repr(sp.decode(sp.encode("ασθενής"))))     # 'ασθενή ⁇ '

Observed output:

vocab_size: 16384
has final sigma ς (U+03C2): False
has medial sigma σ (U+03C3): True
['▁εκ', 'τό', 'ς']
'εκτό ⁇ '
'ασθενή ⁇ '

So the round-trip decode(encode("εκτός")) already loses the «ς» at the tokenizer level,
independent of the acoustic model or decoding strategy.

Why this matters

In Greek orthography a sigma at the end of a word is always written «ς» (never «σ»), so
this affects a very large fraction of Greek words (nominative -ος/-ης/-ες/-ας, genitive -ος,
2nd-person verb -εις, etc.). It makes canary-1b-v2's Greek output orthographically incorrect
everywhere despite the model's otherwise strong Greek acoustic accuracy, and it cannot be fixed
by any runtime flag (see below).

Steps to reproduce

1. pip install -U "nemo_toolkit[asr]" (reproduced on nemo_toolkit 2.7.3, torch 2.6.0+cu124, CUDA 12.4, RTX 3090).
2. Load and transcribe any Greek audio:

   import nemo.collections.asr as nemo_asr
   m = nemo_asr.models.ASRModel.from_pretrained("nvidia/canary-1b-v2", map_location="cuda")
   print(m.transcribe(["greek_clip.wav"], source_lang="el", target_lang="el"))

3. Observe that every word-final «ς» is missing from the output.
4. (Root cause) Run the tokenizer inspection snippet above — «ς» (U+03C2) is not in the vocab.

Expected behavior

Word-final «ς» (U+03C2) is preserved in Greek output, i.e. εκτός, ασθενής, οργανισμός
transcribe with their final sigma intact.

What we ruled out

• Not a NeMo version issue — 2.7.3 is a normal, compatible release; the model card pins no
  version. The behavior is in the tokenizer/model, not the runtime.
• Not a decoding flag — pnc (punctuation/capitalization) and itn (inverse text
  normalization) do not affect letter-level orthography; greedy vs beam makes no difference.
• Not a ς→σ fold — «σ» is emitted correctly elsewhere; the final «ς» is deleted, not
  substituted, because it is genuinely absent from the vocab.

Likely fix (for maintainers)

The final-sigma character «ς» (U+03C2) appears to have been normalized away or excluded during
tokenizer training (or in the Greek training-target normalization), so it never entered the
16,384-piece vocab. A fix would require the tokenizer/vocab to include «ς» (and the model to be
trained/fine-tuned to emit it), or a documented normalization note. As a downstream workaround,
users must post-process Greek output to restore word-final «ς» morphologically, which is
imperfect.

Environment

• Model: nvidia/canary-1b-v2 (snapshot 87bc52657add533cd0156b3fc1aef027280754bf)
• nemo_toolkit: 2.7.3
• torch: 2.6.0+cu124, CUDA 12.4
• GPU: RTX 3090 (also reproduces at the pure-tokenizer level, no GPU needed)
• Language: Greek (el), ASR (source_lang == target_lang)