fix(keyword-matcher): fold Latin diacritics so "PREÇO" matches a "preco" keyword - #37
Merged
diwenne merged 1 commit intoAug 27, 2026
Conversation
…co" keyword
`stripSpecialCharacters` keeps `\p{L}`, which is right, but it means an
accented comment and an unaccented keyword stay different strings. Measured
against the current matcher, all of these miss in BOTH whole-word and partial
mode:
comment "PREÇO?" keyword "preco" miss
comment "ÍNDICE" keyword "indice" miss
comment "informação" keyword "informacao" miss
comment "¿CUÁL es el precio?" keyword "cual" miss
The reverse direction misses too: a campaign keyed on "preço" never matches a
commenter who typed "preco". Accents are typed inconsistently and mobile
keyboards differ, so for Portuguese, Spanish, French and German campaigns this
silently drops real matches, and it drops them quietly because a missed
comment leaves no log row.
`foldDiacritics` normalizes to NFD, drops combining marks, and recomposes.
Applied to both sides inside `matchKeywords`, so it is symmetric.
The fold is deliberately scoped to Latin-script bases, and that scoping is the
substance of this change rather than caution for its own sake. The usual
one-liner:
text.normalize("NFD").replace(/\p{M}/gu, "")
is wrong for a multi-script inbox, because a combining mark carries meaning
outside Latin. Measured against the samples now in the test file:
"किताब" -> "कतब" Devanagari vowel signs deleted
"สวัสดี" -> "สวสด" Thai vowel marks deleted
"مَرْحَبًا" -> "مرحبا" Arabic harakat deleted
"йод" -> "иод" Cyrillic folded to a different word
"Україна" -> "Украіна" Ukrainian ї folded to і
"ガード" -> "カート" Japanese dakuten flipped: "guard" -> "cart"
Walking the decomposed string and dropping a mark only when the base character
before it is `\p{Script=Latin}` leaves every one of those byte for byte
identical, which the tests assert directly.
Input is decomposed before the walk so precomposed (U+00E9) and decomposed
(U+0065 U+0301) sources behave the same; the Instagram API returns both.
Out of scope on purpose: ligature folding. "Fußball" does not match "fussball",
because ß is a letter and not a diacritic, and folding it needs a locale-aware
mapping rather than NFD.
Behaviour that does NOT change: whole-word boundaries still hold after folding
("apreço" does not match a "preco" keyword in whole-word mode), non-Latin
keywords still match as before, `matchedKeyword` is still returned exactly as
the campaign spelled it, and `stripSpecialCharacters` keeps its existing
contract and tests.
Checks: `npm run typecheck`, `npm run lint`, `npm test` (14 files, 158 tests)
and `npm run build` all pass locally. 13 tests added.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
@reisvini is attempting to deploy a commit to the diwenne's projects Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
stripSpecialCharacterskeeps\p{L}, which is correct, but it means an accented comment and an unaccented keyword stay different strings and never match. Measured againstmain(c51f04d), these all miss in both whole-word and partial mode:PREÇO?precoÍNDICEindiceinformaçãoinformacao¿CUÁL es el precio?cualThe reverse direction misses too: a campaign keyed on
preçonever matches a commenter who typedpreco.Accents get typed inconsistently, and mobile keyboards differ on how easy they are to reach, so for Portuguese, Spanish, French and German campaigns this drops real matches. It drops them quietly, because a comment that does not match leaves no log row to notice.
The fix
foldDiacriticsnormalizes to NFD, drops combining marks, recomposes. Applied to both sides insidematchKeywords, so it is symmetric.The fold is scoped to Latin-script bases, and that scoping is the substance of this PR rather than caution for its own sake. The usual one-liner:
is wrong for a tool that has already gone to the trouble of supporting non-Latin scripts, because a combining mark is load bearing outside Latin. Measured, that one-liner does this:
किताबकतबสวัสดีสวสดمَرْحَبًامرحباйодиодУкраїнаУкраінаガードカートWalking the decomposed string and dropping a mark only when the base character before it is
\p{Script=Latin}leaves every one of those byte for byte identical. The tests assert that directly, including the Japanese case.Input is decomposed before the walk so precomposed (
U+00E9) and decomposed (U+0065 U+0301) sources behave the same. The Instagram API returns both.What does not change
apreçodoes not match aprecokeyword in whole-word mode.иодstill does not matchйод.matchedKeywordis still returned exactly as the campaign spelled it, not the folded form, so the DM log and the UI are unaffected.stripSpecialCharacterskeeps its existing contract and its existing tests. The fold is a separate exported function.Out of scope on purpose
Ligature folding.
Fußballdoes not matchfussball, because ß is a letter rather than a diacritic and folding it needs a locale-aware mapping rather than NFD. Happy to do it in a follow-up if you want it.Checks
All four from
CONTRIBUTING.mdpass locally:npm run typecheckcleannpm run lintcleannpm test14 files, 158 tests passing (13 added here)npm run buildsucceeds__tests__/follower-history.test.tsneedsnpm run db:generatebefore it will load, onmainas well as on this branch. Worth a line inCONTRIBUTING.mdif you agree.Why I was in here
I was reading OpenReply closely while planning comment-to-DM automation for a product that ships pt-BR and es-LA as first-class locales, which is how this surfaced. Thanks for open-sourcing it, the Unicode note already in the matcher header is what made the gap easy to spot.