Summary
bb memory search only matches the exact word form typed in the query. In Russian (and other inflected languages) a word has many forms, so a search for сценарий ("script") misses a memory that says сценарии ("scripts"), and the two queries return different result sets. With dozens of memories this makes the search unreliable for non-English users. English is affected too, only less often: migrate does not find migrated.
Versions and environment
- bb 0.46.0, desktop app, macOS 26 (Darwin 25.5.0, arm64)
- Memory plugin 0.2.0 (
builtin:memory)
- Provider does not matter: the search runs in the plugin's server code
Steps to reproduce
bb memory add --scope global --name ru-forms-a --summary "Сценарии роликов писать адаптацией" --details "Сценарии роликов писать адаптацией." --reason repro --json
bb memory search "сценарий" --scope global --json
bb memory search "сценарии" --scope global --json
bb memory search "сценар*" --scope global --json
Expected vs actual
Expected: all three queries return ru-forms-a.
Actual: only the second query (the exact form) returns it. The first returns nothing, and the third also returns nothing because * is stripped by the tokenizer. On my store of 71 global memories, сценарий returned 4 memories and сценарии returned 2 others.
Cause
searchExpression quotes every query token as an exact FTS5 phrase:
|
function searchExpression(query: string): string { |
|
const tokens = query.toLowerCase().match(/[\p{L}\p{N}_-]+/gu) ?? []; |
|
if (tokens.length === 0) |
|
throw new CliError("search query has no searchable terms"); |
|
return [...new Set(tokens)] |
|
.slice(0, 12) |
|
.map((token) => `"${token.replaceAll('"', '""')}"`) |
|
.join(" OR "); |
|
} |
The memories_fts table uses the default unicode61 tokenizer, which does no stemming:
|
CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts USING fts5( |
Proposed fix (prototype works locally)
The fix is on the query side only, with no schema change and no reindex. Each token becomes "token" OR "stem"*:
- Cyrillic tokens are stemmed with the Snowball Russian algorithm. It is a small dependency-free TypeScript module of about 150 lines, and it matches the reference Snowball output on all 49,785 words of
snowball-data/russian/voc.txt.
- Other tokens keep their exact form plus an FTS5 prefix, so
test also finds tests.
- Stems shorter than 4 characters match exactly, so a word like
при does not become a prefix that matches half the store.
ё in the query is folded to е.
Result on the store above: сценарий, сценарии and сценариев now return the same 6 memories. The plugin's existing 13 tests still pass, and I added 9 tests (stemmer forms and CLI search across word forms). I am running it as a fork of the forkable built-in copy.
Known limit: text stored with ё is still not found by a query spelled with е, because unicode61 does not fold ё. Fixing that would need a tokenizer or trigger change, so I left it out of scope.
What I ruled out
Suggested priority and effort
Low effort: about 20 changed lines in server.ts plus one module and tests. This affects every non-English user of the memory plugin. There is no workaround short of guessing word forms in the query, and no data is lost.
I would like to open a PR with the patch if this direction is acceptable.
AGENT GENERATED
Summary
bb memory searchonly matches the exact word form typed in the query. In Russian (and other inflected languages) a word has many forms, so a search forсценарий("script") misses a memory that saysсценарии("scripts"), and the two queries return different result sets. With dozens of memories this makes the search unreliable for non-English users. English is affected too, only less often:migratedoes not findmigrated.Versions and environment
builtin:memory)Steps to reproduce
Expected vs actual
Expected: all three queries return
ru-forms-a.Actual: only the second query (the exact form) returns it. The first returns nothing, and the third also returns nothing because
*is stripped by the tokenizer. On my store of 71 global memories,сценарийreturned 4 memories andсценарииreturned 2 others.Cause
searchExpressionquotes every query token as an exact FTS5 phrase:bb/plugins/memory/server.ts
Lines 300 to 308 in 3912f4a
The
memories_ftstable uses the defaultunicode61tokenizer, which does no stemming:bb/plugins/memory/server.ts
Line 754 in 3912f4a
Proposed fix (prototype works locally)
The fix is on the query side only, with no schema change and no reindex. Each token becomes
"token" OR "stem"*:snowball-data/russian/voc.txt.testalso findstests.приdoes not become a prefix that matches half the store.ёin the query is folded toе.Result on the store above:
сценарий,сценарииandсценариевnow return the same 6 memories. The plugin's existing 13 tests still pass, and I added 9 tests (stemmer forms and CLI search across word forms). I am running it as a fork of the forkable built-in copy.Known limit: text stored with
ёis still not found by a query spelled withе, becauseunicode61does not foldё. Fixing that would need a tokenizer or trigger change, so I left it out of scope.What I ruled out
mainat 3912f4a.plugins/memoryis identical todesktop-v0.46.0.Suggested priority and effort
Low effort: about 20 changed lines in
server.tsplus one module and tests. This affects every non-English user of the memory plugin. There is no workaround short of guessing word forms in the query, and no data is lost.I would like to open a PR with the patch if this direction is acceptable.