[Feature] Add Multilingual Support

[EndeavorYen]

What problem does this solve?

MemPalace currently only works with English. Room classification uses English keyword lists, memory extraction uses English regex patterns, entity detection uses English-only \b[A-Z][a-z]+\b, and spellcheck corrupts CJK text. Non-English users (Chinese, French, Japanese, etc.) get incorrect room assignments, no memory type detection, and broken spellcheck.

The MemChinesePalace fork (https://github.com/Chandler-Sun/MemChinesePalace) shows community demand exists — but it's a full rewrite that drops many features (onboarding, entity registry, palace graph, etc.).

What's the proposed solution?

Use embedding-based semantic classification instead of per-language keyword/regex patterns. The multilingual embedding model (paraphrase-multilingual-MiniLM-L12-v2, ~120MB, CPU-only) understands 50+ languages natively.

Concrete changes:

• Room classification: replace TOPIC_KEYWORDS matching with cosine similarity against room description embeddings. Any language works with zero configuration.
• Memory extraction: replace per-language regex markers with cosine similarity against memory type description embeddings (decision, preference, milestone, problem, emotional). Each paragraph classified independently to preserve multi-type detection.
• Entity detection: add Chinese name patterns (百家姓 surnames, simplified+traditional). This stays rule-based because NER is fundamentally pattern matching, not classification.
• Embedding model: configurable via config.json / env var, default multilingual. sentence-transformers is an optional dependency (pip install mempalace[multilingual]). Core functionality works without it (English regex fallback).
• Spellcheck: auto-skips non-English text via Unicode detection.
• All 7 ChromaDB consumer modules (searcher, miner, convo_miner, mcp_server, layers, palace_graph, cli) use a centralized get_embedding_function() from config.py.

Benchmark results — 173 test cases across 8 languages (zh-Hans, zh-Hant, en, fr, es, de, ja, ko):

Base: 122/122 (100%)    Extended: 51/51 (100%)    219 unit tests, 0 regressions

One new optional dependency: sentence-transformers>=2.0 (install via pip install mempalace[multilingual]). Without it, the system falls back to English regex — existing behavior unchanged. No external API calls. Everything runs locally. No GPU needed.

Alternatives considered:

1. Per-language regex patterns (like MemChinesePalace): Requires ~250 manually curated items per language. Not scalable — adding French/German/Japanese would each need someone to write language-specific keyword lists and regex patterns.

2. LLM-in-the-loop classification: Use Claude/GPT to classify content. Best accuracy but violates MemPalace's "zero API, local only" principle and adds latency/cost.

3. External NLP models (spaCy, etc.): Heavier dependency, larger download, and still language-specific (need per-language model). The sentence-transformer approach is lighter and inherently multilingual.

The embedding approach is the sweet spot: zero per-language config, local-only, lightweight (~120MB model), and 100% benchmark accuracy.

I have a working implementation in my fork: https://github.com/EndeavorYen/mempalace , happy to open a PR if the approach looks good. Also open to splitting into smaller PRs or adjusting based on feedback.

[milla-jovovich]

Hey @EndeavorYen — I've taken a look and ran it through CLI. This is the most architecturally sound answer to the multilingual question in the entire issue queue, and I want to land it.

Your proposal collapses five open issues into one PR: #37 (Chinese community), #50 (cross-lingual search), #92 (French via BGE-M3), #117 (PT-BR), and #156 (PT-BR entity detection). That's the kind of leverage the project needs right now.

On the technical direction:

You're right that the current behavior is English-centric at four different layers:

1. TOPIC_KEYWORDS string matching in room classification
2. Per-language regex for memory type extraction
3. Entity detection via \b[A-Z][a-z]+\b which misses all non-Latin scripts
4. Spellcheck that corrupts CJK text

And you're right that embedding-based semantic classification is the correct fix for (1) and (2) — cosine similarity against room/memory-type description embeddings generalizes to any language the embedding model supports, with zero per-language code. That's a pattern I hadn't considered and it's genuinely elegant.

For (3), your point that NER is fundamentally pattern matching and should stay rule-based is correct, and adding 百家姓 (Baijiaxing) surname tables for Chinese (simplified + traditional) is the right shape. Same reasoning for Japanese / Korean — rule-based with language-specific tables.

For (4), auto-skipping non-English text via Unicode detection is the cleanest fix.

On benchmark results:

Your 173 test cases across 8 languages (zh-Hans, zh-Hant, en, fr, es, de, ja, ko) is genuinely useful validation. I'd want those numbers published in the PR description so contributors can see what the fix moves for each language. And I'd want the benchmark itself shipped as tests/test_multilingual_classification.py or similar, so it becomes a regression test.

Proposed PR shape:

• mempalace[multilingual] as an optional extra — users who only speak English don't pay the 120MB download cost
• sentence-transformers as an optional dependency
• paraphrase-multilingual-MiniLM-L12-v2 as the default multilingual model (configurable via mempalace.yaml / env var)
• Centralized get_embedding_function() in config.py — all 7 ChromaDB consumer modules (searcher, miner, convo_miner, mcp_server, layers, palace_graph, cli) route through it
• Graceful English-only fallback when the multilingual extra isn't installed
• The 173-test benchmark suite added to tests/
• Documentation for installing + switching languages

One coordination point:

This overlaps with #273 (@valentt's 10-agent production setup uses Qwen3-Embedding-8B via Ollama — which is the top-MTEB multilingual model). Worth making the embedding function pluggable enough that Qwen3 via Ollama drops in as a config change, not a code change. Your get_embedding_function() already enables this shape; just make sure the config surface supports Ollama endpoints.

What this does to #37, #50, #92, #117, #156:

Once your PR lands, I'll close #50, #92, #117, #156 as "folded into #231 multilingual support." #37 stays open as the pinned Chinese community thread — the fix lands there but the thread lives on as the community home.

Want to open the PR? Happy to review when it's up. This is the highest-leverage single action in the queue right now.

Thanks for the thoughtful write-up + the benchmark work. 💜

[giocaizzi]

I wonder how this would work for my case, where for some projects I work in a given language (prompting and actual code, variable and filenames) and sometimes in other projects, as I do not have that restriction, I work in English but happen to have conversations in my mother language - not English.

Happy to test this!