Match keywords typed in Arabic script, and stop splitting combining marks

[hhakamian82]

Two bugs that share one line of the matching pipeline. Both are reproduced by the tests in this PR; every assertion added here returned false before the change.

1. stripSpecialCharacters destroys combining marks before foldDiacritics can fold them

stripSpecialCharacters keeps \p{L} and \p{N} and turns everything else into a space. A combining mark is neither, so marks became spaces before foldDiacritics ran — which defeats the diacritic folding this module exists to provide:

stripSpecialCharacters("señor".normalize("NFD"))  // "sen or"
matchKeywords("señor".normalize("NFD"), ["senor"]) // false

The module's own docstring notes that "Instagram returns both" precomposed and decomposed forms, so this is reachable in production, not a theoretical case. The same replace split Devanagari and Thai vowel signs off their base letters ("किताब" → "क त ब").

Fix: add \p{M} to the keep-set. Marks now survive to foldDiacritics, which already decides per script which ones are safe to drop.

2. Arabic-script text never matches across keyboard layouts

foldDiacritics deliberately leaves non-Latin marks alone, because they are load bearing in Devanagari, Thai and Japanese. In the Arabic script they are not — and neither are several letter variants.

A commenter typing Persian on an Arabic keyboard sends U+064A (ي) and U+0643 (ك) where the account owner typed U+06CC (ی) and U+06A9 (ک). The strings render identically and never compare equal, so a campaign keyed on لینک misses every one of those commenters. This is not an edge case for a Persian-speaking account; both layouts are in everyday use.

Cases that were all silently missing:

comment	keyword	before	after
لينك بده	لینک	miss	match
كمك	کمک	miss	match
کد۵	کد5	miss	match
قیمت‌ها	قیمتها	miss	match
لیــنک	لینک	miss	match
لِینک	لینک	miss	match

normalizeArabicScript folds yeh/kaf/alef/teh-marbuta variants, drops harakat, tatweel and ZWNJ, and maps Persian and Arabic-Indic digits to ASCII. It is applied to both sides of the comparison, so it never matters which form the account owner typed into the campaign builder. Latin, Cyrillic and CJK are byte-for-byte untouched — there is an assertion for that.

ZWNJ is deleted rather than turned into a space because the no-separator spelling is the fallback people actually type.

Behaviour preserved

• Whole-word matching still holds in Persian: لینکدونی does not match لینک with wholeWordMatch, and does with partial matching.
• Existing keyword-matcher.test.ts is unchanged and still passes.

Verification

npm test        184 passed (16 files)
npm run typecheck   clean
npm run lint        clean

🤖 Generated with Claude Code

[vercel]

@hhakamian82 is attempting to deploy a commit to the diwenne's projects Team on Vercel.

A member of the Team first needs to authorize it.

[diwenne]

What this does — 2 fixes, 1 pipeline line:

1. stripSpecialCharacters was splitting combining marks (\p{M}) before foldDiacritics could fold them.
Kept \p{L} + \p{N}, turned everything else into a space. A combining mark is neither, so NFD "señor" (e+´) became "sen or" and never matched "senor". Same split Devanagari/Thai vowel signs ("किताब" -> "क त ब"). Fix: keep \p{M} — marks survive to foldDiacritics, which already knows per-script which to drop (Latin yes, Devanagari/Thai/Japanese no).

2. Arabic-script campaigns never matched across keyboard layouts.
foldDiacritics intentionally leaves non-Latin marks alone. In Arabic script those marks/variants are the same letter: Persian ی/ک vs Arabic ي/ك (U+06CC vs U+064A, U+06A9 vs U+0643) render identically, plus alef/teh-marbuta variants, harakat, tatweel, ZWNJ, and Persian/Arabic-Indic digits. A campaign keyed on لینک missed every Arabic-keyboard لينك comment. New normalizeArabicScript folds all of those on both sides of the comparison (yeh->ی, kaf->ک, alef variants->ا, teh marbuta->ه, harakat/tatweel/ZWNJ/bidi deleted, Eastern digits->ASCII). ZWNJ is deleted not spaced — "قیمت‌ها" vs "قیمتها" have to be equal because half of Instagram types it, half does not.

Safe for existing users? Yes. Idempotent? Yes.

• Existing users (Latin/Cyrillic/CJK/emoji): No behavior change except bugfix. The \p{M} repair makes decomposed "señor"/"preço" match their plain keywords as documented — precomposed forms already worked. Latin fold still scoped to Latin bases only (Cyrillic й/ї, Japanese ガード, Devanagari किताब untouched — all covered by foldDiacritics tests). normalizeArabicScript regexes only hit Arabic-script codepoints; "Клод link 链接" is byte-for-byte identical (asserted). Whole-word boundaries preserved ("لینکدونی" still does not match "لینک" in whole-word mode). Existing keyword-matcher.test.ts is unchanged.
• Idempotency / rollback: Pure in-memory normalization at match time, no schema/data migration, no persisted state. Applied symmetrically so f(f(x)) == f(x) and keyword vs comment form does not matter. Deterministic string replace — running twice is the same as once. Rolling back just re-introduces the missed-match false negatives, no data corruption.
• Verified locally: npm test 168 -> 184 passed (16 files), typecheck clean, lint clean. All 16 new assertions were false before the change.

Merging.

[diwenne]

Reviewed locally: 184/184 tests pass, typecheck/lint clean. \p{M} fix repairs NFD diacritic folding; normalizeArabicScript is narrowly scoped to Arabic-script codepoints, applied both sides, no effect on Latin/Cyrillic/CJK. Safe + idempotent — no migration, pure match-time normalization.