Skip to content

fix(keyword-matcher): fold Latin diacritics so "PREÇO" matches a "preco" keyword - #37

Merged
diwenne merged 1 commit into
diwenne:mainfrom
reisvini:fix/keyword-matcher-latin-diacritics
Aug 27, 2026
Merged

fix(keyword-matcher): fold Latin diacritics so "PREÇO" matches a "preco" keyword#37
diwenne merged 1 commit into
diwenne:mainfrom
reisvini:fix/keyword-matcher-latin-diacritics

Conversation

@reisvini

Copy link
Copy Markdown

The bug

stripSpecialCharacters keeps \p{L}, which is correct, but it means an accented comment and an unaccented keyword stay different strings and never match. Measured against main (c51f04d), these all miss in both whole-word and partial mode:

comment keyword result
PREÇO? preco miss
ÍNDICE indice miss
informação informacao miss
¿CUÁL es el precio? cual miss

The reverse direction misses too: a campaign keyed on preço never matches a commenter who typed preco.

Accents get typed inconsistently, and mobile keyboards differ on how easy they are to reach, so for Portuguese, Spanish, French and German campaigns this drops real matches. It drops them quietly, because a comment that does not match leaves no log row to notice.

The fix

foldDiacritics normalizes to NFD, drops combining marks, recomposes. Applied to both sides inside matchKeywords, so it is symmetric.

The fold is scoped to Latin-script bases, and that scoping is the substance of this PR rather than caution for its own sake. The usual one-liner:

text.normalize("NFD").replace(/\p{M}/gu, "")

is wrong for a tool that has already gone to the trouble of supporting non-Latin scripts, because a combining mark is load bearing outside Latin. Measured, that one-liner does this:

input naive fold what broke
किताब कतब Devanagari vowel signs deleted
สวัสดี สวสด Thai vowel marks deleted
مَرْحَبًا مرحبا Arabic harakat deleted
йод иод different Cyrillic word
Україна Украіна Ukrainian ї folded to і
ガード カート dakuten flipped: "guard" becomes "cart"

Walking the decomposed string and dropping a mark only when the base character before it is \p{Script=Latin} leaves every one of those byte for byte identical. The tests assert that directly, including the Japanese case.

Input is decomposed before the walk so precomposed (U+00E9) and decomposed (U+0065 U+0301) sources behave the same. The Instagram API returns both.

What does not change

  • Whole-word boundaries still hold after folding: apreço does not match a preco keyword in whole-word mode.
  • Non-Latin keywords match exactly as before, and иод still does not match йод.
  • matchedKeyword is still returned exactly as the campaign spelled it, not the folded form, so the DM log and the UI are unaffected.
  • stripSpecialCharacters keeps its existing contract and its existing tests. The fold is a separate exported function.

Out of scope on purpose

Ligature folding. Fußball does not match fussball, because ß is a letter rather than a diacritic and folding it needs a locale-aware mapping rather than NFD. Happy to do it in a follow-up if you want it.

Checks

All four from CONTRIBUTING.md pass locally:

  • npm run typecheck clean
  • npm run lint clean
  • npm test 14 files, 158 tests passing (13 added here)
  • npm run build succeeds

__tests__/follower-history.test.ts needs npm run db:generate before it will load, on main as well as on this branch. Worth a line in CONTRIBUTING.md if you agree.

Why I was in here

I was reading OpenReply closely while planning comment-to-DM automation for a product that ships pt-BR and es-LA as first-class locales, which is how this surfaced. Thanks for open-sourcing it, the Unicode note already in the matcher header is what made the gap easy to spot.

…co" keyword

`stripSpecialCharacters` keeps `\p{L}`, which is right, but it means an
accented comment and an unaccented keyword stay different strings. Measured
against the current matcher, all of these miss in BOTH whole-word and partial
mode:

    comment "PREÇO?"              keyword "preco"        miss
    comment "ÍNDICE"              keyword "indice"       miss
    comment "informação"          keyword "informacao"   miss
    comment "¿CUÁL es el precio?" keyword "cual"         miss

The reverse direction misses too: a campaign keyed on "preço" never matches a
commenter who typed "preco". Accents are typed inconsistently and mobile
keyboards differ, so for Portuguese, Spanish, French and German campaigns this
silently drops real matches, and it drops them quietly because a missed
comment leaves no log row.

`foldDiacritics` normalizes to NFD, drops combining marks, and recomposes.
Applied to both sides inside `matchKeywords`, so it is symmetric.

The fold is deliberately scoped to Latin-script bases, and that scoping is the
substance of this change rather than caution for its own sake. The usual
one-liner:

    text.normalize("NFD").replace(/\p{M}/gu, "")

is wrong for a multi-script inbox, because a combining mark carries meaning
outside Latin. Measured against the samples now in the test file:

    "किताब"  -> "कतब"       Devanagari vowel signs deleted
    "สวัสดี"   -> "สวสด"       Thai vowel marks deleted
    "مَرْحَبًا"  -> "مرحبا"      Arabic harakat deleted
    "йод"    -> "иод"       Cyrillic folded to a different word
    "Україна" -> "Украіна"   Ukrainian ї folded to і
    "ガード"   -> "カート"      Japanese dakuten flipped: "guard" -> "cart"

Walking the decomposed string and dropping a mark only when the base character
before it is `\p{Script=Latin}` leaves every one of those byte for byte
identical, which the tests assert directly.

Input is decomposed before the walk so precomposed (U+00E9) and decomposed
(U+0065 U+0301) sources behave the same; the Instagram API returns both.

Out of scope on purpose: ligature folding. "Fußball" does not match "fussball",
because ß is a letter and not a diacritic, and folding it needs a locale-aware
mapping rather than NFD.

Behaviour that does NOT change: whole-word boundaries still hold after folding
("apreço" does not match a "preco" keyword in whole-word mode), non-Latin
keywords still match as before, `matchedKeyword` is still returned exactly as
the campaign spelled it, and `stripSpecialCharacters` keeps its existing
contract and tests.

Checks: `npm run typecheck`, `npm run lint`, `npm test` (14 files, 158 tests)
and `npm run build` all pass locally. 13 tests added.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Aug 27, 2026

Copy link
Copy Markdown

@reisvini is attempting to deploy a commit to the diwenne's projects Team on Vercel.

A member of the Team first needs to authorize it.

@diwenne
diwenne merged commit e1b13ca into diwenne:main Aug 27, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants