Forked for production hardening — happy to discuss upstream

[hang-in]

Hi jaytoone,

I really liked the retrieval algorithm in CTX and forked it to harden the Claude Code hook implementation for production use:

https://github.com/hang-in/tunaCtx

Retrieval logic is unchanged. The work focused on:

• Decomposing bm25-memory.py (1837 LOC) into orchestrator + 11 sub-modules under _bm25/
• Unifying BM25 tokenizer/scoring between eval and production paths
• Adding 82 unit tests + 26 golden fixture regression guards
• Safety hardening (sqlite_vec fallback, deterministic sort, atomic install with hash-based update)

Original copyright (MIT) is preserved in LICENSE, and the README explicitly credits this as a fork.

If any of these changes look useful for upstream, I'd be happy to extract them as PRs. No pressure either way — happy to keep it as a downstream fork too.

Thanks for the great tool.

[pluto2060]

This is really impressive work — decomposing bm25-memory.py into 11 sub-modules with 82 unit tests is exactly what it needs. The original file grew organically and the 1837-LOC monolith is a real maintenance problem.

I would love to pull this upstream. A few questions before I open a tracking PR:

1. Tokenizer unification — which file controls the canonical tokenizer in your fork? That is the change I am most interested in merging first.
2. sqlite_vec fallback — is this a graceful degradation (BM25-only mode on import failure) or a full replacement?
3. Hash-based update in atomic install — is this in ctx_retriever/cli/install.py or a separate module?

Happy to take the 11 sub-modules as-is if you want to open a PR — or I can cherry-pick specific pieces. Either way works.

Thank you for preserving the MIT attribution and for the kind framing.

[hang-in]

Hi jaytoone,

Glad the audit was useful. To your three questions:

1. Tokenizer unification

Canonical entry point: src/hooks/_bm25/tokenizer.py (function tokenize). It does:

• Korean particle strip (handful of common 조사: 은/는/이/가/을/를/에/의/와/과/로/도/만)
• Lowercase + Porter stem (English)
• Stopword removal (combined Korean + English short list)

Callers that go through it post-unification:

• src/hooks/_bm25/ranker.py (BM25 scoring primitives — score_corpus_bm25, bm25_rank_decisions)
• src/retrieval/adaptive_trigger.py
• src/retrieval/bm25_retriever.py
• src/evaluator/coir_evaluator.py
• benchmarks/eval/doc_retrieval_eval_v2.py

Integration sanity check is scripts/verify_bm25_unified.py — verifies all callers route through _bm25/tokenizer.tokenize and produce identical token streams for a fixed input. The archival benchmark scripts (g1_*.py, g2_*.py, mab_*.py, nemotron_*.py) deliberately keep their own BM25 implementations to preserve historical A/B comparison points — I'd suggest leaving those alone.

Note (deliberate follow-up, not in this unification PR): the current tokenizer is NL-oriented — Porter stem + Korean particle strip. It's weak on code identifiers — getUserById is treated as a single token, and the _ in snake_case_function isn't a word boundary, so a query for ""user"" can't match getUserById. The natural next step is a subtoken splitter (camelCase / snake_case / dot.notation), but I'd keep that as a separate cycle. The unification PR prioritizes behavior-identical migration and carries today's tokenization as-is. (Not yet implemented in our fork either — flagging it for visibility.)

This is the cleanest piece to land first — single tokenizer file plus a few caller edits, no behavior change (token stream is byte-identical; BM25 score deltas are 0 on the regression suite).

2. sqlite_vec fallback

Graceful degradation, not a replacement.

Location: src/hooks/chat-memory.py:16

try:
    import sqlite_vec
    HAS_VEC0 = True
except ImportError:
    HAS_VEC0 = False

Downstream behavior:

• If HAS_VEC0: vault.db is opened with the vec0 extension loaded; chat-memory does FTS5 + vec0 hybrid (α=0.5 cosine + 0.5 BM25)
• Otherwise: falls back to FTS5-only search; the hook still produces output, just without dense rerank
• The hook also emits a one-line warning so users know the semantic layer is off

Motivation: keeps chat-memory functional on environments without sqlite_vec wheels (CI containers, fresh Windows installs, conda envs without a compile toolchain). The previous unguarded import sqlite_vec would silently turn the entire chat-memory hook into a no-op on those.

3. Hash-based update in install

Module: src/cli/install.py (the full ctx-install entry point lives here).

Split helpers:

• src/cli/settings_patcher.py — atomic write to ~/.claude/settings.json (temp file + os.replace) with timestamped .backup_<TS>.json per write. Idempotent: re-running ctx-install does not create duplicate hook entries.
• The hash-based update lives inside install.py itself: SHA256 of source vs. installed hook → if different, copy + create .backup_<TS>.py of the previous version.

Three new flags in install.py:

• --force-hooks — skip hash compare, overwrite unconditionally
• --no-update-hooks — never overwrite (preserve user edits)
• --uninstall — remove ctx-install's settings.json entries (hook files left in place by default; --force also removes them)

Test coverage: tests/unit/test_settings_patcher.py (22 tests) + tests/unit/test_install_cli.py (32 tests). Real filesystem rename is exercised, not mocked.

On PR shape

If cherry-picking sounds easier than pulling the whole decomposition, here's a split that lands cleanly with each piece independent:

1. Tokenizer unification (Q1) — smallest, safest first move. No public API change, behavior-identical.
2. sqlite_vec graceful degradation (Q2) — tiny patch, prevents the hook from going silently dark.
3. install.py hash-based update + atomic settings_patcher (Q3) — the install machinery; useful even without the rest.
4. bm25-memory.py decomposition into the 11 sub-modules — the largest piece; depends on (1) being landed since modules import the canonical tokenizer.
5. Telemetry instrumentation — opt-in via env var, zero-cost when off.

Outside the 5-piece scope (separate cycle candidate, not part of these PRs):

• Code-identifier subtoken splitting in the tokenizer — see the follow-up note under Q1 above. Not implemented in our fork yet either; keeping it separate preserves the deterministic <100ms posture of the unification PR. If you're already exploring this direction, I'd happily fold our work into a separate PR or issue rather than overlap.

Happy to open them in that order with the relevant unit tests each, or as one bundled PR if you'd rather review the whole thing at once. Both options pass golden + 105 unit tests against current upstream master.

Let me know which shape fits your review bandwidth best.

Thanks again for the open door on this.

[pluto2060]

Hey hang-in — this is exactly the kind of hardening CTX needs. Thank you for the detailed writeup.

I looked at tunaCtx and the work is substantial. A few specific things I'd love to pull upstream:

High priority for PRs:

1. Tokenizer unification — the eval/production BM25 scoring divergence is a known gap (it was causing misleading benchmark numbers). This is the one I want most.
2. 82 unit tests + 26 golden fixtures — CTX's CI is thin right now and this would directly fix that.
3. Deterministic sort — bm25-memory.py has a subtle non-determinism bug I've been meaning to address.

Already in 0.3.14:

• sqlite_vec fallback — landed this week after catching the hard import crash on fresh installs.

On the decomposition (1837 LOC → 11 sub-modules): I'd want to discuss the boundary design before merging — the monolithic file is a deliberate tradeoff (single-file hooks are easier to copy/audit), but your approach might be worth it if the module boundaries are clean.

Please open PRs for whichever pieces you're happy to extract — I'll review promptly. Even partial cherry-picks are welcome. And if you want to co-maintain upstream, that's also on the table.

Thanks again for preserving the MIT license and attribution.

[hang-in]

Hey jaytoone — thanks for the detailed triage. Here's a status update with PRs opened (Draft) and four points worth flagging from the audit cycle that produced them.

Three Draft PRs opened

PR	Mapping to your priorities
#3 — [Tokenizer] Unify _bm25.tokenizer canonical entry across eval pipeline	P0 (the one you wanted most)
#4 — [Docs Search] Dedup root README/CLAUDE/MEMORY against same-name docs/research/ files	P1 audit + an unrelated production-hook bug discovered along the way
#5 — [Determinism] Explicit tiebreak on 3 ranker.py sort sites	P2 (the "subtle non-determinism bug" you flagged)

All three are Draft because the change sites live inside the _bm25/ package layout (PR-4 territory, awaiting your boundary review). Once you've had a chance to look at the boundaries, I can either (a) cherry-pick onto the merged decomposition branch, or (b) re-author each change directly inside the upstream monolith and discard these Drafts. Either path is fine on my end.

I dropped sqlite_vec fallback from the original 5-stage plan since it's already in 0.3.14 (ba7df3d).

Four findings worth flagging from the audit

1. Your 08e262b already covers part of PR-1

Your fix: Korean tokenizer gap in eval pipeline + 6 regression tests (with the Related: hang-in/tunaCtx tokenizer.py attribution — appreciated) already addressed doc_retrieval_eval_v2.py. PR #3 covers the remaining three sites we found in the eval pipeline (g1_docs_bm25_eval.py, g1_longterm_baseline_eval.py, g2_docs_paraphrase_eval.py). Two further sites (src/cli/telemetry.py, src/retrieval/bm25_retriever.py) are intentionally kept divergent — rationale annotated inline (telemetry is identifier-frequency stats not BM25 ranking; bm25_retriever needs raw TF on code identifiers, and canonical's dict.fromkeys() dedup would flatten that).

2. Test count was 82, audit settled it at 80 + 26 golden

I quoted "82 unit tests + 26 golden" in the original issue body. After classifying tests for upstream relevance:

• 80 unit tests (upstream-ready: test_trigger_classifier_ko.py 45, test_ast_improvements.py 19, test_chat_memory_fallback.py 9, test_code_search_sort.py 7)
• 23 unit tests gated on PR-4 decomposition (test_bm25_init_reexport.py, test_bm25_memory_cache.py, test_bm25_memory_telemetry.py)
• 66 unit tests fork-only (atomic install machinery — test_install_cli.py, test_settings_patcher.py, test_uninstall_cleanup.py)

So the test-suite PR (when it comes) ships 80 + 26 golden, not 82 + 26. Apologies for the ±2 drift in the original number.

3. PR #4 carries an unrelated production-hook bug fix

While re-running the golden suite I caught a real bug: build_docs_bm25 indexes docs/research/*.md and root extras (CLAUDE.md, README.md, MEMORY.md) without name-collision dedup. When docs/research/README.md exists alongside root README.md, both get indexed under the same name, and the path that returns bm_filtered[:top_k] without rerank can emit both — surfaces as a duplicate > README.md line in the G2-DOCS output. The fix in PR #4 switches corpus build to a name-keyed dict with root extras winning on collision. Originally PR-2 was just the test suite; this audit finding got bundled in.

4. PR #5 ships 5 regression cases for determinism

The three ranker.py sort sites previously relied on CPython's stable-sort guarantee for equal-key ordering. PR #5 adds explicit tiebreak keys (matching the existing pattern at code_search.py:233) plus tests/regression/test_pr3_deterministic_sort.py covering: idempotency, equal-rank tiebreak independent of input order, equal-score tiebreak by hash, no-emb sanity, index tiebreak.

Co-maintain — yes, happy to

I use CTX daily so the maintenance overhead is already paid on my end. Concretely I can take:

• Issue triage on weekday afternoons KST
• Hook reviews (BM25 / tokenizer / install machinery / golden fixtures)
• Release notes + version bumps in coordination with you

I'd defer to you on retrieval algorithm direction, paper alignment, and anything benchmark-facing — those are your design decisions and I'd like to keep upstream's voice intact. If a MAINTAINERS.md with explicit areas of ownership works for you, I can draft one as a follow-up after the boundary discussion settles.

Order of operations I'd suggest

1. You take a look at the _bm25/ boundary in PR [Tokenizer] Unify _bm25.tokenizer canonical entry across eval pipeline (Draft, P0 of #1) #3 (or any of the three — same _bm25/ layout) and let me know whether it's broadly OK or needs redrawing.
2. Once boundaries are agreed, I either land PR-4 (decomposition) first and the three Drafts become clean diffs, or I re-author each into the upstream monolith and close the Drafts.
3. Subtoken splitting stays out of all four PRs (separate cycle, as agreed earlier in this thread).

Let me know if the order or scope needs adjusting.

[pluto2060]

Hey hang-in — this is the most thorough audit writeup I've gotten on CTX, and it has already shipped concrete improvements. Here's where things stand after today:

What just landed (commit 7f9d5a9)

PR #4 (dedup) and PR #5 (determinism tiebreaks) are both ported directly into bm25-memory.py — your implementation details for PR #5 (function names, the rrf_merge dict-insertion-order risk) made the port clean. You're credited in the commit message.

On the _bm25/ boundary question

Decision: monolith path for now. The decomposition in tunaCtx is the right long-term architecture, but I don't want to ship the package boundary in the same cycle as the functional fixes — too much blast radius if something breaks in production hooks. So:

• PRs [Tokenizer] Unify _bm25.tokenizer canonical entry across eval pipeline (Draft, P0 of #1) #3/[Docs Search] Dedup root README/CLAUDE/MEMORY against same-name docs/research/ files (Draft) #4/[Determinism] Explicit tiebreak on 3 ranker.py sort sites (Draft, P2 of #1) #5 as "re-author into the upstream monolith" (not cherry-pick onto decomposition branch)
• The _bm25/ package layout conversation stays open as a separate tracked item (I'll open a new issue for it)

For PR #3 (tokenizer): yes please — re-author the 3 remaining eval sites (g1_docs_bm25_eval.py, g1_longterm_baseline_eval.py, g2_docs_paraphrase_eval.py) directly into the upstream paths. The 2 intentionally divergent sites (telemetry.py, bm25_retriever.py) — your annotation rationale is exactly right, leaving those as-is.

On the test count

80 + 26 golden noted — I'll update any docs that cite 82. The classification (upstream-ready vs gated vs fork-only) is exactly how I'd have split it.

Co-maintenance — yes, very much yes

This is the part I'm most interested in. Your offer:

Issue triage on weekday afternoons KST, hook reviews, release notes

That's exactly the gap that exists right now. I'd draft MAINTAINERS.md as you suggested, with the ownership split you outlined:

• You: BM25/tokenizer/install machinery/golden fixtures, issue triage, release notes
• Me: retrieval algorithm direction, paper alignment, benchmark decisions

Let's do it — open a MAINTAINERS.md PR whenever you're ready and I'll merge it. No need to wait for the boundary discussion.

Suggested order of operations (updated)

1. You re-author PR [Tokenizer] Unify _bm25.tokenizer canonical entry across eval pipeline (Draft, P0 of #1) #3 tokenizer fix into upstream monolith → I merge
2. New issue for _bm25/ package boundary (separate track, no rush)
3. Test-suite PR (80 + 26 golden) when boundary is clear
4. MAINTAINERS.md in parallel — unblocked

Thank you again for all of this.

[hang-in]

Thanks for the clear direction — agreeing with all of it.

Plan from this side

1. PR [Tokenizer] Unify _bm25.tokenizer canonical entry across eval pipeline (Draft, P0 of #1) #3 re-author into upstream monolith — opening a new PR shortly with the 3 eval sites (g1_docs_bm25_eval.py, g1_longterm_baseline_eval.py, g2_docs_paraphrase_eval.py) wired to call bm25-memory.py's tokenize() directly via importlib.util (same loader pattern as the re-skinned PR-5 regression). The 2 intentionally divergent sites (telemetry.py, bm25_retriever.py) stay annotated as-is.
2. MAINTAINERS.md — opening that PR in parallel today, since you said no need to wait for the boundary discussion.
3. _bm25/ boundary — happy for that to live in a separate tracked issue. No rush from this side either.
4. Test-suite PR (80 + 26 golden) — held until the boundary track lands, as you suggested.

Two small notes

On commit 7f9d5a9 — flagged this on PR #5 about a minute after your issue comment, so the timing crossed in the air: from my side git fetch upstream master shows HEAD at efa88be, and the GitHub API returns 422 on 7f9d5a9. build_docs_bm25 and the three ranker sort sites still look pre-fix. Looks like the local commit may not have made it past git push — totally not urgent, just flagging in case it's a missed sync. I'll close PRs #4 and #5 once the port shows up upstream.

On the credit — appreciated. Whether it lands as 7f9d5a9 or whatever SHA the eventual push gets, the credit framing in the commit message is the part that matters, and that's already exactly what I was hoping for.

Co-maintenance

Yes, very much in. Drafting MAINTAINERS.md with the ownership split as we discussed:

• hang-in: BM25 / tokenizer / install machinery / golden fixtures, issue triage (weekday afternoons KST), release notes
• jaytoone: retrieval algorithm direction, paper alignment, benchmark decisions

I'll keep it minimal — areas of ownership, decision protocol for cross-area changes, contact, and a one-line note on how the boundary discussion (item 3 above) will be revisited. Tag you on the PR.

Thank you again for the clean review on all three.

[pluto2060]

The 7f9d5a9 push was missing — now synced to master. Both PR #4 and #5 fixes are visible upstream.

For your end:

• PR [Tokenizer] Unify _bm25.tokenizer canonical entry across eval pipeline (Draft, P0 of #1) #3 (tokenizer re-author) and MAINTAINERS.md — opening both today sounds great, I'll review and merge same-day.
• The _bm25/ boundary as a separate tracked issue — I'll open it this session.

The MAINTAINERS.md framing you've outlined is exactly right. Looking forward to the PR.