fix(rewrite): correct multibyte UTF-8 compound rewrites (#383)

[mpecan]

Closes #383

Summary

Compound shell commands that mixed a multibyte UTF-8 character (✓, ✗, emoji, CJK) in one segment with a chain operator (&&/||/;/newline) were rewritten into invalid shell — spurious |/; tokens were spliced at the wrong byte offsets, so the command failed to parse. This broke every tokf hook-driven rewrite that mixed an emoji/checkmark echo with a filterable pipe.

Before / after

in:   echo "✓ valid" || echo "✗ bad"; ls ~/.claude/skills/ | grep tokf
was:  echo "✓ valid" | |echo "✗ bad";tokf run --baseline-pipe 'grep tokf' ls ~/.claude/skills/   ← invalid
now:  echo "✓ valid" || echo "✗ bad"; tokf run --baseline-pipe 'grep tokf' ls ~/.claude/skills/   ← valid

Root cause

Two layers:

1. Upstream (rable): AST spans are character indices (the lexer scans a Vec<char>), but span ends were computed as pos + value.len() — a char position plus a byte length. For multibyte input the end overshot, so even Node::source_text returned the wrong slice for non-final tokens. Fixed in fix(lexer): compute span ends by character count, not byte length rable#72, released as rable 0.2.1.
2. tokf: compound_segments/flatten_segments sliced the byte-indexed source string by raw (character) span values. Now routed through a char_to_byte helper that mirrors rable's own conversion.

The issue's suspected culprit (rules.rs:75, the {rest} slice) was a red herring — regex offsets are byte offsets by design.

Changes

• char_to_byte helper + the two separator-slice sites in rewrite/bash_ast.rs.
• Bump rable 0.1.15 → 0.2.1 (absorbs the 0.2.0 API surface changes; tokf only uses rable::parse + rable::ast).

Tests

Three layers of regression coverage — all previously failing, now green:

• AST round-trip + exact separators — new bash_ast_multibyte_tests.rs (split out to stay under the 700-line file limit).
• In-process rewrite — tests_compound.rs.
• Real binary an LLM hook invokes — cli_rewrite.rs.

cargo test -p tokf (989 lib + 59 CLI), clippy, and fmt all clean.

🤖 Generated with Claude Code

[repository-butler]

Filter Verification Report

Changed Filters

No filter files changed in this PR.

All Filters Summary

✅ 143/143 test cases passed across 51 filters

---

_{Generated by tokf verify}