idl-gen: unlexable.rs scanner misses reserved-prefix shapes and desyncs on multi-byte/c-string literals (follow-up to #253)

[vpavlin]

Context

PR #253 adds spel-framework-core/src/idl_gen/unlexable.rs, a byte-level scanner that skips path-dep source files containing a macro metavariable glued to a string literal (the ark-ff 0.3.0 stringify!($Fp"({})") shape), because re-lexing that shape under edition 2021 inside the generate_idl! proc macro queues a fatal prefix is unknown diagnostic.

The PR fixes the reported real-world trigger, but empirical testing (standalone runs of has_metavar_glued_literal cross-checked with rustc --edition 2018 vs --edition 2021) shows the detection has holes in both directions. Filing them here as agreed follow-up rather than blocking the merge.

1. Missed glued shapes (fatal diagnostic still comes back)

Edition 2021 reserves any identifier glued to ", ', or # — not just $metavar + ". All three of these compile with zero errors under --edition 2018 and fail with prefix `...` is unknown under 2021, yet the scanner returns false for each:

macro_rules! m { ($f:ident) => { stringify!($f'a') } }   // prefix `f` is unknown
macro_rules! m { ($f:ident"px") => { () } }              // prefix `ident` is unknown
macro_rules! m { () => { stringify!(foo"bar") } }        // prefix `foo` is unknown

A pre-2021 path dependency containing any of these still aborts the consumer's build — the original #253 failure mode, un-fixed for these shapes.

2. Multi-byte char literal desyncs the scanner (can mask the target shape itself)

skip_char_or_lifetime uses a fixed 3-byte lookahead (b[i + 2] == b'\''), so a multi-byte char literal like 'é' makes the scanner land inside the next char literal and flip string/code parity for the rest of the file:

let v = ['é','"']; stringify!($Fp"({})")   // scanner returns false — the exact ark-ff shape is missed

So the fix can silently fail on its own target shape depending on surrounding code.

3. cr#"..."# raw C-strings not recognized (false positive skips a clean file)

Raw C-string literals (Rust 1.77+) aren't handled: the leading c sets prev_ident, gating off the raw-string arm, so the raw body is scanned by skip_string with escape semantics. A body ending in a backslash (e.g. a Windows path) desyncs quote parity:

let p = cr#"C:\temp\"#; let s = "cost $USD"; #[account_type] pub struct A { x: u8 }
// scanner returns true — clean file skipped, its #[account_type] items silently dropped from the IDL

This is the inverse failure: valid modern code loses account types from generated IDLs with only a stderr warning.

Possible directions

• Harden the scanner in place: generalize detection to any identifier/metavar glued to a quote (excluding the legal r/b/br/c/cr prefixes), fix skip_char_or_lifetime for multi-byte chars, recognize c"…"/cr#"…"# literals, and add the six cases above as regression tests.
• Replace shape detection: route the dep-file parse through proc-macro2's fallback lexer (e.g. parsing where proc_macro::is_available() is false) so lex errors return as ordinary Err instead of queuing fatal diagnostics — covers the whole class and deletes the scanner, but the mechanism needs validation inside the proc-macro context.

Related smaller items noted in the same review: the skip is applied unconditionally, so the spel generate-idl CLI path (which uses proc-macro2's fallback lexer and parses these files fine) now also drops items it previously harvested; and the doc comment on generate_idl_from_file_with_deps says scan problems "arrive as warnings" but unreadable files and syn parse failures return silently without warning.