Clarify which text normalizations are allowed for custom tokenizers

[mikeapedia]

Context

PR #1578 (Custom Casefold Tokenizer) applies .lower() after NFKC normalization before tokenization, which has prompted a productive debate about what normalizations are permitted. The discussion is happening in the PR comments, but the question is broader than any single submission and deserves its own policy thread.

The question

Custom tokenizers are explicitly allowed (Issues #43, #897). The rules state: "Instead of locking the tokenizer, we let you bring your own and calculate our validation metrics on the average compression of the validation set."

But there's no guidance on what text normalization is permitted before or during tokenization. Every SentencePiece submission currently applies nmt_nfkc normalization, which was inherited from SP defaults rather than an explicit competition policy.

Why this matters

Any normalization that collapses distinct representations reduces prediction entropy and makes the modeling task easier. This is a spectrum:

Normalization	What it collapses	Info loss
NFC (canonical only)	Encoding variants of same character (e + ◌̈ → ë)	None — lossless
NFKC (current default)	Ligatures, fractions, superscripts, fullwidth chars, math symbols + whitespace collapsing + newline conversion	Lossy
NFKC + case folding (PR #1578)	Above + uppercase → lowercase	More lossy
Degenerate (e.g., all text → "a")	Everything	Total

The competition currently uses NFKC — a lossy normalization — inherited from SentencePiece defaults. Reasonable people can disagree on where to draw the line. The current debate in PR #1578 has surfaced good arguments on both sides.

Proposal

Could maintainers clarify:

1. Is NFKC the intended normalization standard, or just an inherited default? If submissions used NFC instead, would that be allowed? What about identity (no normalization)?

2. Are additional normalizations beyond NFKC permitted? Case folding, accent stripping, and other lossy normalizations all sit on the same spectrum as NFKC.

3. Should the policy be documented? A one-line addition to the README or rules would save future submitters (and reviewers) from re-litigating this for each new tokenizer PR.

Whatever the decision, explicit documentation would benefit the competition. I'll respect whatever the maintainers decide.

[SPThole]

In my view, imagine in inference setting, wouldn’t this reduce the effectiveness of a generative model—particularly with transformations like uppercasing → lowercasing?

This kind of normalization makes more sense for encoder/classifier setups, where semantics matter more than surface form. However, for generative models, casing and formatting are part of the output quality.

Removing that distinction during training means the model can’t learn or reproduce it later, which may hurt usability even if it improves compression.

[sharpobject]

Applying any sort of unicode normalization is not lossless.

In my view, imagine in inference setting, wouldn’t this reduce the effectiveness of a generative model—particularly with transformations like uppercasing → lowercasing?

This is also true of the nmt_nfkc normalization already applied, which prevents the model from ever training on the "_" character or on spans of more than 1 space.

Should the policy be documented? A one-line addition to the README or rules would save future submitters (and reviewers) from re-litigating this for each new tokenizer PR.

Yes, ideally the README can be updated to say that the exact bytes of each document in the the validation set must be reproducible by decoding the token ids or whatever.

[Qeuph]

Hmm