[sean0407]

Summary

Preserve embedded NUL characters in the C tokenizer by using an out-of-range codepoint for tokenizer boundaries and treating NUL as ordinary input text.

Evidence

• Reproduced the issue on current main: the C tokenizer returned 'a' for a\x00b, while the pure-Python tokenizer returned the full input.
• Root cause: Tokenizer_read() returned \\0 for end of input and the C marker table also classified \\0 as an end marker, so a real NUL stopped parsing.
• The boundary sentinel now cannot collide with valid Unicode input. Added tokenizer fixtures for embedded NUL in plain text, a longer string, and a template.
• Checks run: isolated C-extension build passed; pytest -q tests --ignore=tests/test_docs.py passed (2,011 tests); the C and Python tokenizer outputs match for all three reproductions.

Merge Danger

Door: two-way

Blast Radius: localized