MCP server silently serves with stale library versions after package upgrade

[messelink]

Summary

Long-running MCP server processes cache mempalace and chromadb in sys.modules at startup and never reload. Users who upgrade mempalace or its dependencies mid-session continue serving tool calls with stale code, leading to silent data corruption or API mismatch errors. Claude Code users regularly hit this because a single Claude Code session (and its dedicated MCP server) can live for days or weeks via session resume.

Problem scenario

We experienced this during the chromadb 0.6.x → 1.5.7 transition enabled by PR #690 (which removed the <0.7 upper bound):

1. Claude Code session started on day 1 with mempalace + chromadb 0.6.x — MCP server imports both into memory
2. User runs pipx install --force -e . on day 5 (or any other upgrade path: pip install -U mempalace, uv pip install -U, etc.) — pipx venv now has chromadb 1.5.7
3. The already-running MCP server is oblivious — sys.modules['chromadb'] is still pinned to 0.6.x in memory
4. User continues issuing mempalace_add_drawer / mempalace_diary_write calls via MCP
5. Those writes go through the 0.6.x code path and produce database rows with legacy format (e.g., seq_id as BLOB instead of INTEGER)
6. A fresh process (CLI mining, or a newly-started Claude Code session) reads those rows and crashes during compaction with errors like:

   chromadb.errors.InternalError: Error in compaction: Error reading from metadata segment reader:
   error occurred while decoding column 0: mismatched types;
   Rust type `u64` (as SQL type `INTEGER`) is not compatible with SQL type `BLOB`
   

   or

   'dict' object has no attribute 'dimensionality'
   

There is no warning or error at tool-call time in the stale MCP server — it reports success. The inconsistency is only discovered later, when a fresh process hits the legacy rows.

Impact

• Silent data inconsistency across multi-day Claude Code sessions
• Bug amplification: stale-library writes contribute to the compaction crashes that PR fix: batch upserts in miner to prevent ChromaDB 1.5.x compaction crashes #796 is trying to mitigate on the miner side
• Multi-session blast radius: one user can have 3+ Claude Code sessions running (on different projects, or via remote-wrapped MCP like SSH-to-raindance). Each session's dedicated MCP server has its own staleness timer, and all of them write to the same palace.
• Cross-version silent merges: a single palace can accumulate rows written by different mempalace+chromadb versions over days, with no audit trail or warning.
• Existing mitigations don't help: mempalace migrate detects read compatibility but not write/compaction compatibility (fix: mempalace migrate produces incomplete ChromaDB 0.6.x schema #722). PR feat: post-migration schema validation for mempalace migrate #735 addresses the schema validation gap after migration, but a stale in-memory process bypasses migration entirely.

Reproducing

1. pipx install -e . — note the installed mempalace and chromadb versions
2. Start Claude Code and make any MCP tool call (forces MCP server startup → imports)
3. On the host, pipx install --force -e . or pip install -U chromadb — upgrade in the same venv
4. In the existing Claude Code session, call mempalace_add_drawer — succeeds using stale library
5. Start a new Claude Code session or run mempalace mine from the CLI — crashes on the first write that touches metadata touched by the stale process

Suggested fixes (not mutually exclusive)

A. Version check at tool-call time (minimum viable fix)

On MCP server startup, record the running versions:

import importlib.metadata
_STARTUP_MEMPALACE_VERSION = importlib.metadata.version(\"mempalace\")
_STARTUP_CHROMADB_VERSION = importlib.metadata.version(\"chromadb\")

On each tool call, compare against the currently installed versions. If they differ, return a clear error:

{
  \"error\": \"MCP server library is stale (started with mempalace X.Y.Z / chromadb A.B.C, disk has X.Y.W / chromadb A.B.D). Restart the MCP server to pick up the new version.\",
  \"action_required\": \"restart_mcp_server\"
}

B. Refuse writes on mismatch

If any version mismatch is detected, disable write tools (add_drawer, diary_write, kg_add, mine via MCP, etc.) but keep read tools functional. Prevents further inconsistency while still letting the user read their palace.

C. Self-restart via os.execv()

More aggressive: on detecting staleness, os.execv(sys.executable, [sys.executable, \"-m\", \"mempalace.mcp_server\"]) to replace the current process with a fresh Python. Risk: loses any in-flight state, not always safe in Python processes with threads or C extensions (chromadb has Rust bindings).

D. Documentation-only mitigation (until a code fix lands)

Update README, plugin docs, and PR #340's description with a note about restarting MCP servers after any mempalace or chromadb upgrade. This is what Claude Code users currently need to do manually.

Related

• PR fix: add mempalace-mcp entry point for pipx/uv compatibility #340 — adds mempalace-mcp console entry point making the MCP server actually work for pipx/uv users. This bug was latent before fix: add mempalace-mcp entry point for pipx/uv compatibility #340 because pipx/uv users couldn't run the MCP server at all. Our fix unlocks the door.
• PR fix: remove chromadb <0.7 upper bound — blocks 1.x installs #690 — removed chromadb<0.7 upper bound, creating the version mismatch gradient users can cross
• Issue fix: mempalace migrate produces incomplete ChromaDB 0.6.x schema #722 / PR feat: post-migration schema validation for mempalace migrate #735 — post-migration schema validation. Fixes a related but different gap (migrate command completeness), not the in-process staleness
• PR fix: batch upserts in miner to prevent ChromaDB 1.5.x compaction crashes #796 — batch upserts in miner to prevent compaction crashes. Mitigates the downstream symptom, not the root cause