From 49307822be1f52310f7f219a355761624996aa24 Mon Sep 17 00:00:00 2001 From: Zack Jackson <25274700+ScriptedAlchemy@users.noreply.github.com> Date: Thu, 5 Nov 2026 22:19:14 -0800 Subject: [PATCH 1/2] docs: remove stale code storage design passages --- docs/LEAN-CTX-COMPARE.md | 2 +- docs/V2-OPERATING-MODEL.md | 2 +- docs/dashboard.md | 4 +- docs/graph-at-rest/README.md | 151 +--- docs/plans/storage/README.md | 70 +- docs/plans/storage/evidence/why/inv-3-docs.md | 4 +- docs/plans/tracedecay-v2/00-plan-set-index.md | 27 +- docs/plans/tracedecay-v2/05-query-crate.md | 7 +- ...ality-evaluation-and-retrieval-research.md | 2 +- ...cross-project-repository-worktree-scope.md | 4 +- .../25-code-intelligence-indexing-crate.md | 2 +- .../26-observability-accounting-and-usage.md | 2 +- ...1-native-fastembed-semantic-code-search.md | 708 ------------------ ...8-storage-retention-size-and-efficiency.md | 11 +- .../39-embedded-grafeo-graph-database.md | 127 +--- ...generation-storage-and-zero-copy-replay.md | 60 -- 16 files changed, 61 insertions(+), 1122 deletions(-) delete mode 100644 docs/plans/tracedecay-v2/31-native-fastembed-semantic-code-search.md diff --git a/docs/LEAN-CTX-COMPARE.md b/docs/LEAN-CTX-COMPARE.md index 80d39b1052..5ea251f4ca 100644 --- a/docs/LEAN-CTX-COMPARE.md +++ b/docs/LEAN-CTX-COMPARE.md @@ -12,7 +12,7 @@ through its daemon and product contract. | Concern | TraceDecay | lean-ctx | |---|---|---| | Code and evidence | A daemon-owned authority serves code, project-memory, and session evidence with exact provenance, generation, coverage, and receipts. | Describes a local context layer for read modes, shell output, request-context compression, and context-use visibility. | -| Graph and durable state | Grafeo owns admitted graph/vector data; SQLite owns relational/content records. Clients do not open either store directly. | Describes a property graph and portable context packages alongside its context-management features. | +| Graph and durable state | Grafeo owns admitted graph data; SQLite owns relational/content records. Clients do not open either store directly. | Describes a property graph and portable context packages alongside its context-management features. | | Session context | Lossless Context Memory retains raw-message evidence and summary nodes, with bounded replay and drill-down through daemon operations. | Describes persistent session memory and recovery across chats. | | Workflow and multi-agent work | The V2 product includes the daemon-owned work graph and typed workflow runtime: task evidence, ownership, authorization, execution, cancellation, recovery, collaboration, and receipts are product data, not host scripts. | Describes agent handoff and shared-context features. | | Operating model | Hosts submit bounded hints and typed operations; daemon convergence, freshness, and maintenance remain explicit. | Describes agent wrappers, shell hooks, and an optional request proxy that can mediate context on the read and model-request paths. | diff --git a/docs/V2-OPERATING-MODEL.md b/docs/V2-OPERATING-MODEL.md index 62d812a2f8..d11149e147 100644 --- a/docs/V2-OPERATING-MODEL.md +++ b/docs/V2-OPERATING-MODEL.md @@ -10,7 +10,7 @@ capabilities are available. ## Authorities - `tracedecay-graph-db` is the only TraceDecay boundary for the workspace-pinned - embedded Grafeo `=0.5.42` runtime. It exposes typed TraceDecay identities, + embedded Grafeo runtime. It exposes typed TraceDecay identities, operations, snapshots, generations, and outcomes; Grafeo handles and types do not escape it. - Grafeo is in-process and embedded. It has no graph server, sidecar, network diff --git a/docs/dashboard.md b/docs/dashboard.md index 9ebc37ea9b..b84cfb6d71 100644 --- a/docs/dashboard.md +++ b/docs/dashboard.md @@ -25,8 +25,8 @@ repository and generation provenance, coverage, and receipts. Warming, partial, unavailable, denied, cancelled, retained, and recreation-required outcomes are distinct UI states, never a blank successful panel. -Structure views present Grafeo-backed graph and vector results supplied by the -daemon. Conversation and memory views hydrate authorized content from its owning +Structure views present Grafeo-backed graph results supplied by the daemon. +Conversation and memory views hydrate authorized content from its owning authority. Work controls separate proposal, authorization, execution, and repository effects. Diagnostics are read-only; maintenance actions are explicit daemon operations with authorization and receipts. diff --git a/docs/graph-at-rest/README.md b/docs/graph-at-rest/README.md index 5a17164514..d0ad8837a2 100644 --- a/docs/graph-at-rest/README.md +++ b/docs/graph-at-rest/README.md @@ -1,9 +1,5 @@ # An at-rest form for the graph: measurement and verdict -Bench-lane investigation. Nothing here is wired into production; the two -harnesses are `#[ignore]`d and the one new crate method is gated on -`test-helpers` so it cannot compile into a shipped binary. - ## The question Activating a sealed generation replays every node and edge into grafeo's in-RAM @@ -31,17 +27,8 @@ This investigation measured what calling it would buy. > and served read-only. That dissolves blocker 2 (nothing ever writes to a > sealed artifact), the pinned fork's compact-store property hash index > dissolves blocker 1 (sealed reopens register `INDEXED_PROPERTIES` exactly -> like staging opens), and vector search deliberately never routes to sealed -> stores. Bytes round-trip the pinned dictionary codec losslessly now, but -> `COMPACT_ROUND_TRIPS_BYTES` stays **off**: a real 431-file code generation -> (~45k entities carrying serialized-record Bytes payloads) fails its -> post-reopen recovered-digest proof in compact form with "relation scalar -> endpoints do not match native topology" and stays permanently unseated -> behind activation retries, while the same corpus seals and serves in replay -> form; the toy-scale sealed-store contract compacts and reads exactly, so -> the defect only shows at generation scale. This document remains the -> measurement record for the whole-database compaction that was measured and -> rejected. +> like staging opens). This document remains the measurement record for the +> whole-database compaction that was measured and rejected. > > **Refreshes seal as layers (graph format 7):** a small refresh no longer > re-encodes its whole generation. Its sealed store is a delta container @@ -56,18 +43,6 @@ This investigation measured what calling it would buy. > becomes the next base. Stores sealed under format 6 are superseded and > rebuild cold; no migration exists. -## Verdict - -**`compact()` cannot be applied to TraceDecay's graph as it is currently -shaped.** Not "does not pay off", it returns an error. - -The blocker is TraceDecay's own schema, not grafeo. And under a schema that -does not trip it, the columnar form is a large, real win: **10.9x faster -activation at 1M rows, 2.4x lower resident memory, 185x faster adjacency.** - -So the at-rest work is worth doing, but the first ticket is a schema change, -not a storage change. - ## Numbers All figures from a single box, `--test-threads=1`, one scenario per process @@ -308,11 +283,6 @@ close/reopen verification. Small: the crate already holds `compact_snapshot_for_bench` does today. Everything downstream (section write, section reload, layered rewiring) is already implemented in grafeo. -**Phase 3, restore vector-index planning.** Either override -`has_vector_index` on the layered store so it consults the overlay and the -vector section, or hoist the check in `src/runtime.rs` off `graph_store()`. -Small, but do not skip it: gap 3 fails silently. - **Phase 4, optional fork work.** Streaming section serialize (gap 5) and mmap-backed container open (gap 6). Both are pure wins on top of a working path, and neither is needed for the activation number. @@ -424,104 +394,6 @@ contract failures, a sealed generation reopened twice came back with its format marker gone, which is the same defect reached through TraceDecay's own lifecycle rather than the minimal probe. -### The vector index is durable now, fork branch ready, pin not moved - -The claim above that the HNSW index "was never durable across a reopen -anyway" was true, and it turned out to be a wiring gap rather than a -missing format. - -**Where durability was lost.** grafeo has always *written* the index. -`build_sections` emits a `SectionType::VectorStore` carrying each index's -full HNSW topology, entry point, max level, per-node per-level neighbour -lists, and `CatalogSection::collect_indexes` records every index's -label, property, dimensions, metric, M and ef_construction beside it. The -restore side dropped both: - -- `CatalogSection::deserialize` handed the loader its `property_indexes` - and nothing else. The `vector_indexes` it had just decoded were - discarded (`catalog_section.rs`). -- `load_from_sections` guarded the topology restore on - `store.vector_index_entries()` being non-empty - (`database/mod.rs`, the `SectionType::VectorStore` block). A cold open - builds its `LpgStore` from nothing, so that map is empty at exactly the - moment the guard runs. The section bytes were read off disk and - dropped on the floor. - -So the index was durable on disk and absent in memory, and every reopen -re-indexed the whole corpus. - -**Fork, not sidecar.** The section format round-trips HNSW faithfully -already: topology is all it needs to carry, because the vectors live in -LPG node properties (persisted anyway) and are read through a -`VectorAccessor` at query time. Nothing was missing from the format, only -the restore wiring, a tracedecay-side sidecar would have duplicated a -working serializer to re-derive an artifact grafeo already had on disk, -and left every other consumer broken. - -**Branch:** `tracedecay/0.5.42-vector-index-durable`, stacked on -`tracedecay/0.5.42-close-and-overlay`, at -`f38218653dfc69fda67f9c669371036fde1ed5fe`. Three commits: - -1. re-register catalog definitions as empty indexes and let the section - fill their topology. Two incomplete-restore cases are deliberately - left absent for rebuild rather than half-restored, a definition with - no section, and a definition the section carried no topology for, - because an index that exists and covers nothing reports - `has_vector_index() == true` and answers every search with nothing. - Quantized indexes are left to a rebuild too: their codebook lives - inside the index and the section carries only topology, so restoring - one would silently downgrade it to full precision. The catalog - snapshot is v2 to record the mode; v1 files still load. -2. mark index creation, removal, and binding changes as unlogged state. - `close` may skip its flush when the WAL has not moved, and nothing - about an index reaches the WAL, so an index built after open never - reached the file, and the next open rebuilt it, and the next. Harmless - while indexes were always rebuilt; load-bearing the moment they are - restored. -3. price it: 5,000 x 64-dim vectors, release profile, **8.7ms** to - restore the index as part of opening the whole database against - **359.6ms** to rebuild the index alone on a database already open. - ~41x, and identical neighbour sets across 24 probes. - -The branch also adds an opaque per-index **binding token** that rides the -catalog section, so a caller can stamp the generation digest its index -was built over and compare after a reopen instead of trusting that a -restored topology still describes the rows. - -**The pin has moved.** The vector-durability commits were reintegrated -onto the fork's `fix/catalog-version-guard` branch (ported over the -out-of-place checkpoint and close-skip storage rewrite that landed in -the meantime) and the five `grafeo-*` revs in `[patch.crates-io]` now -point at that lineage. The reopen tests in `tests/graph_db_suite/runtime_contract.rs` -assert `Available`, restored, within the same admission bound that -proves no rebuild ran, and the fork gates the semantics with -`vector_index_reopen`, `vector_index_restore_cost`, and -`torn_vector_checkpoint` (a checkpoint killed at every injection point -over a populated HNSW index must reopen and serve search). - -### `has_vector_index`, resolved, and worse than gap 3 described - -Gap 3 said `CompactStore` inherits the trait default `false`. The layered store -is more specific than that: `LayeredStore::has_vector_index` delegates to the -*overlay* (`compact/layered.rs:1081`), which `compact()` leaves empty, so it -answers `false` for indexes the store had a moment earlier. That part is at -least truthful, the HNSW index really is gone, and it was never durable across -a reopen anyway. - -The silent failure is one layer along. `create_vector_index` builds by scanning -`nodes_by_label(label)`, and TraceDecay's vector label is -`entity_projection_label`, one of several labels on an entity node. On a -compacted base that scan matches nothing, so the build succeeds over zero -vectors, `has_vector_index` then answers `true`, and every vector search -returns empty while reporting an index exists. - -The resolution, when phase 2 becomes unblocked, is the routed one: refuse to -build an HNSW index over a store serving from a columnar base, with a typed -error rather than an empty index, and keep a generation that carries vector -indexes in its replay form. Both halves were implemented against the seal-time -compaction and reverted with it; neither is reachable while nothing calls -`compact()` outside the bench lane. - ### Why "never write after compact" is not a guarantee TraceDecay can make The obvious containment for blocker 2 is to promise the store is never mutated @@ -563,19 +435,10 @@ the heap-copy path and are a floor, not a ceiling. every reopened store ran without them. That was invisible while identity resolved through the intrinsic label index, and cost 23.7s for 64 point reads at 500k once it did not. -- **Phase 2, seal implies compact. Landed as per-generation stores; compact - form still gated.** The whole-database `compact()` this plan scoped stays - rejected; the sealed-store lane (`src/sealed_store.rs`) seals each - generation into its own single-generation database, exactly the - "per-generation stores" design blocker 2 demanded, and compacts it when - the rows carry no Bytes or Vector properties. The fork's compact property - index settles blocker 1's economics and the pinned dictionary codec - round-trips Bytes losslessly, but Bytes-carrying generations stay in replay - form behind `COMPACT_ROUND_TRIPS_BYTES` until a generation-scale compact - seal passes its post-reopen proof (see the constant's doc for the measured - scale failure). -- **Phase 3, vector planning. Not applicable to sealed stores:** vector - search never routes to a sealed artifact; HNSW indexes are rebuilt against - the staging database (`apply_sealed_copy_batch` doc). +- **Phase 2, seal implies compact. Landed as per-generation stores.** The + whole-database `compact()` this plan scoped stays rejected; the sealed-store + lane (`src/sealed_store.rs`) seals each generation into its own + single-generation database, exactly the "per-generation stores" design + blocker 2 demanded. - **Phase 4, streaming section writer, mmap container open.** Unchanged, and now clearly behind a property index for the columnar base in priority. diff --git a/docs/plans/storage/README.md b/docs/plans/storage/README.md index 67e644e9a0..e1ad159665 100644 --- a/docs/plans/storage/README.md +++ b/docs/plans/storage/README.md @@ -158,14 +158,10 @@ this redesign does not need to settle; the last cutover step measures it. initialized subdirectory of a worktree (a monorepo subproject, which today's `discover_project_root` supports), the pair of root commit and relative path. On this host 7 of the 14 tracedecay clones have a local-path origin, 5 https and - 2 ssh, so only the root commit converges all of them; the 14 clones and 32 linked - worktrees resolve to one project. + 2 ssh; the 14 clones and 32 linked worktrees resolve to one project. - A project never re-keys. An unshallowed clone, a first commit in an unborn repository, or a new root after a fetch adds an alias to the existing project. So durable memory is never stranded under an old id. -- `tracedecay project link ` and `tracedecay project split` record an - explicit alias or refusal in `catalog.db`, scoped to this profile, for forks that - should not share and unrelated repositories that should. - A cwd outside any repository and not explicitly initialized belongs to `projects/_profile/`. It never mints a project of its own. - Code indexes stay per worktree in `worktrees//`. Segments are @@ -275,9 +271,9 @@ in. reconnect. Batched writes take one savepoint per job, so a job's ordinary error rolls back only that job. A suspect-connection error rolls back the whole batch; each job of the batch then runs alone on a fresh connection. -- One fixed pool, `tracedecay-store-N` (default 6, clamp 2 to 16), serves every - file. A lane is per-file state: a write queue (the existing deficit-round-robin - `FairQueue`), a read queue, one write connection, at most two read connections, +- One fixed pool, `tracedecay-store-N`, serves every file. A lane is per-file + state: a write queue (the existing deficit-round-robin `FairQueue`), a read + queue, one write connection, at most two read connections, and fence state. Any pool thread can run any lane; at most one thread holds a lane's write connection at a time. One thread is reserved for foreground classes (catalog, journal, memory) so bulk derivation cannot starve Work. Open lanes are @@ -368,24 +364,10 @@ Each growth source from #4099: Each file carries `user_version` and a schema digest. The rule follows the class: - authoritative and durable classes (catalog, journal, memory, evidence): - forward-only, additive migrations in the crate, applied by the executor at open. - A non-additive change copies into `.next` and renames over the file; no - copy is kept. No user-authored state is reset by a schema change. + admit only the exact final shape. Any other shape returns typed + `ResetRequired`; there is no migration. - rebuildable classes (derived, code): a mismatch means unlink and rebuild. -Upgrade path for beta installs: `tracedecay wipe --stale` plus `tracedecay init` -remains the reset for anything this design does not carry. Durable classes are -carried, not reset. Because the store key stays `ProjectId` and the directory stays -`projects//`, each project converts in place the first time the new -binary opens it: `memory_v2_*` rows and the anchors they cite move from -`tracedecay.db` to `memory.db`, workflow definitions and native-integration -approvals move from `sessions.db` to `journal.db`, configuration revisions move from -`global.db` to `catalog.db`, and the old files are deleted after the copy commits. -Evidence is re-ingested from host files rather than copied. The conversion runs once -per file, is deleted two releases later, and is not a compatibility reader: no code -reads the old files after the conversion. The operator host was wiped on 2026-10-16 -and needs none of it. - ### 8. Public port surface | Type | Public methods | @@ -512,7 +494,8 @@ Act on, and the change made: `forget` deletes durable data. - Evidence and journal were reset or unlinked on a schema change, destroying LLM summaries, history past host retention, workflow definitions and approvals (A, B, - C). All authoritative and durable classes now migrate forward; corruption salvages. + C). Authoritative and durable classes now admit only the exact final shape, and + corruption salvages. - Facts cite retrieval anchors through a foreign key, so anchors could not live in evictable evidence (B). Cited anchors live in `memory.db`. - `FnOnce` jobs made the reconnect-and-rerun ladder and fence reruns impossible, @@ -549,15 +532,11 @@ blocking pool; that hop is intentional and the executor step removes it. byte on disk. - We accept losing cross-page snapshot consistency, because every read is a short job, in exchange for a total checkpoint fence. Pagination uses keyset cursors. -- We accept that forks sharing a root commit share a project until `project - split`, in exchange for clones converging with no configuration. - We accept partial derived search during a derived rebuild, in exchange for corruption and schema changes that heal without an operator. - We accept a 30 s ceiling per write job; bulk work pages. - We accept Grafeo as an immutable file format. This is not a second engine: it has no WAL and no mutation, like the code-index generation files. -- We accept a one-shot in-place conversion of durable rows for two releases, - because "durable memory survives" forbids a reset of that class. ## Alternatives considered @@ -588,10 +567,8 @@ changes a file that holds memory. it, and no read parks a native worker. Proof: a native task awaiting a typed read leaves the native reader-wait counter unchanged (a mutation that reads inline fails the test). -2. Memory durability. The stores that hold memory run at FULL; the memory tables - get additive migrations instead of `ResetRequired` with their first schema - change. Proof: those stores' writers report `synchronous=FULL` and the others - `NORMAL`. +2. Memory durability. The stores that hold memory run at FULL. Proof: those + stores' writers report `synchronous=FULL` and the others `NORMAL`. 3. Growth stops now. Worktree liveness by root instead of the shared `.git`, a collector for unregistered directories that never deletes durable data, response-handle sweeps every tick, `hook_analytics.jsonl` rotation, age-pruned @@ -618,14 +595,12 @@ changes a file that holds memory. `ProfileLost` within one tick with the descriptors closed. 8. Partition the sessions file and identity by repository. Split each project into journal, memory, evidence and derived; telemetry leaves the catalog; identity - adopts through aliases. The in-place conversion of durable rows ships here. - Per-stream cursors and dead letters replace the global queue: row poison - dead-letters per stream, and corrupt derived authority (ghost or divergent - provenance) rebuilds `derived.db` instead of being skipped. Delete + adopts through aliases. Per-stream cursors and dead letters replace the global + queue: row poison dead-letters per stream, and corrupt derived authority (ghost + or divergent provenance) rebuilds `derived.db` instead of being skipped. Delete `projection_queue`, `lcm-payloads/`, the string classifier, the store locator resolver and standalone registries. Proof: two full clones and a linked worktree - resolve to one project with three worktree directories; converted facts, - workflow definitions and approvals read back identically; a poisoned row + resolve to one project with three worktree directories; a poisoned row dead-letters while other streams project; corrupt provenance triggers a derived rebuild and the stream resumes. 9. Graph narrowing. The memory graph and session-relation graphs move to SQLite @@ -647,14 +622,11 @@ changes a file that holds memory. - Plan 02's "No asynchronous database API" and its interactive repository transactions: every storage API is async and every transaction is one job. -- Plan 38's "Store soft budgets are owner-configured and inert by default": budgets - have defaults and are enforced by `Storage::maintain`. +- Store soft budgets have defaults and are enforced by `Storage::maintain`. - Plan 39's move of graph-shaped data off SQLite: mutable graph data (memory and session relations) lives in SQLite; Grafeo remains an immutable sealed format. - Plan 40's monolithic per-worktree generation segments: segments are content-addressed per project. -- The roadmap's reset-only rule for released stores, for the durable and - authoritative classes: they migrate forward; derived and code still rebuild. - The 2026-07-15 fold of sessions into the project-wide store, which had no recorded reason: ADR-003's failure-domain split is restored by class. @@ -675,9 +647,7 @@ changes a file that holds memory. profile memory store (`user-memory.db`), commit at `synchronous=FULL`; every other store keeps `NORMAL`. `global.db` stays `NORMAL` although the catalog class is FULL, because it still takes per-hook analytics writes until telemetry splits out - in step 8. The additive-migration rule for memory ships with the first change to a - memory table (the conversion in step 8), not as machinery with no migration to - run. The class contract is unchanged. + in step 8. The class contract is unchanged. - Step 3 (2026-10-16, done). Four PRs stopped the growth this step names: - #4120. Orphan-store liveness no longer counts the shared git common directory, so a deleted worktree's store becomes collectable while its @@ -1021,11 +991,3 @@ changes a file that holds memory. get it; it is deferred until a silent-corruption incident is on record. - Are 64 GiB per profile and 8 GiB of evidence per project the right defaults? Step 10 calibrates them from the first weeks of persisted usage. - -## Next implementation step - -The rest of step 5: the native writer-wait counter becomes a refusal of any -synchronous writer wait on a native thread and both counters are deleted; then -the interactive exact-SQL transactions (56 direct sites and about 190 behind the -runtime-core async `Transaction`) become jobs. Step 7's first slice, writers and -readers as tasks on one fixed pool, is in progress in parallel. diff --git a/docs/plans/storage/evidence/why/inv-3-docs.md b/docs/plans/storage/evidence/why/inv-3-docs.md index 822c5db144..ce9449411a 100644 --- a/docs/plans/storage/evidence/why/inv-3-docs.md +++ b/docs/plans/storage/evidence/why/inv-3-docs.md @@ -107,13 +107,13 @@ cross-references, "page history" is `git log`/pickaxe on `docs/`. 9. **Stated goal: move graph-shaped data off SQLite and custom adjacency structures.** - "**Goal:** Replace custom adjacency structures and graph-shaped SQLite storage with one embedded Grafeo runtime boundary while retaining SQLite only for genuinely relational, transactional, and content-bearing records." - - Where: `docs/plans/tracedecay-v2/39-embedded-grafeo-graph-database.md` line 12, unchanged since the first version 955552d44b (ScriptedAlchemy, 2026-08-03, "docs(v2): plan embedded Grafeo graph cutover"). The code commit 0e3284dd2f ("feat(query): embed Grafeo graph projection") landed earlier the same day. All of these commit messages have empty bodies. + - Where: `docs/plans/tracedecay-v2/39-embedded-grafeo-graph-database.md` goal, unchanged since the first version 955552d44b (ScriptedAlchemy, 2026-08-03, "docs(v2): plan embedded Grafeo graph cutover"). The code commit 0e3284dd2f ("feat(query): embed Grafeo graph projection") landed earlier the same day. All of these commit messages have empty bodies. - Plan index (00, lines ~1003-1013): "graph-shaped data moves off custom adjacency structures and SQLite. Relational journals, receipts, raw content, configuration, and execution fencing remain in SQLite." - Relevance: the only stated "why". No doc compares Grafeo against SQLite recursive CTEs or adjacency tables on a measured workload before adoption. Task 12 ("Compare against the pre-Grafeo baseline") remained unchecked, and the 2026-08-14 audit lists "E18 POST-RC Grafeo aggregate/performance provenance ... pre-Grafeo performance comparison" as outstanding (deleted `audits/v2-implementation-audit-2026-08-14.md`). 10. **Plan 40 states what the pre-V2 SQLite graph did well and what V2 adds.** - "Master (pre-V2) proved the opposite trade-offs work. SQLite adjacency graph, per-file incremental sync, no stored body text, open-and-serve startup, but lacked what V2 genuinely adds: immutable digest-sealed generations, verify-then-publish with a watermark, exact branch/revision identity. Doc 39 already assigns the durable graph to Grafeo; the cutover never completed, leaving the sealed JSON as both canonical authority **and** de facto serving store." - - Where: `docs/plans/tracedecay-v2/40-partitioned-generation-storage-and-zero-copy-replay.md` §Problem (status "ACTIVE (decided 2026-08-22)"). + - Where: `docs/plans/tracedecay-v2/40-partitioned-generation-storage-and-zero-copy-replay.md` §Problem. - Measured trigger in the same doc: "One sealed code generation for this repository is a single `generation-.json` of **1.39 GB**"; "Warmup on this repository read tens of GB, pinned the 8 GiB cgroup boundary, and starved queries." - Relevance: the doc credits immutable generations and verify-then-publish to V2's design. It credits the earlier SQLite graph with good operational economics. diff --git a/docs/plans/tracedecay-v2/00-plan-set-index.md b/docs/plans/tracedecay-v2/00-plan-set-index.md index 72e8ce80b2..20bc503d1d 100644 --- a/docs/plans/tracedecay-v2/00-plan-set-index.md +++ b/docs/plans/tracedecay-v2/00-plan-set-index.md @@ -106,13 +106,9 @@ invariants and do not by themselves keep a journey open. receipt-owned paths and a drift-converges acceptance test. Operational closure still requires the measured doctor journey and a clean Cursor install/upgrade/doctor pass on an operator machine.) -- Semantic search is disabled by an invalid configuration snapshot. Plan 20 - owns final-snapshot validity and explicit reset/recreation; Plan 31 owns - semantic activation. - Exact, lexical, and graph retrieval remain the required available fallback. - This snapshot does not mean semantic search is product-disabled. - (Superseded 2026-09-13: dense semantic retrieval is retired by rejected - decision 11; there is no activation to repair.) +- Dense semantic retrieval is retired by rejected decision 11. Exact, lexical, + and graph retrieval remain the required available authorities; there is no + semantic activation to repair. - A live profile was observed 237 minutes stale (the index later reported 285 minutes stale while this reconciliation was being checked). Plan 25 and the active incremental-indexing slice own cadence/freshness diagnosis; the @@ -275,8 +271,8 @@ records the rejected mechanism, the reason, and the retained replacement: installation downloads no model, loads no inference runtime, builds no vectors, and exposes no activation workflow. Released relational stores retain their data under the admission exception above; dense retrieval - is never reactivated. Plan 31 is retired as an authority; its lexical-fallback - and exact-tier obligations already belong to Plans 05, 15, and 25. The + is never reactivated. The lexical-fallback and exact-tier obligations + already belong to Plans 05, 15, and 25. The retained replacement is exactly three code-intelligence authorities and no fourth "semantic" one: lexical search (exact, fielded, phrase, proximity, bounded identifier/path splitting, and named query-time aliases that stay @@ -932,8 +928,7 @@ Unmeasured speculative optimizations and placeholder benchmarks do not ship. [15](15-search-quality-evaluation-and-retrieval-research.md), [23](23-session-lcm-temporal-retrieval-and-evaluation.md), and [25](25-code-intelligence-indexing-crate.md). - [Plan 31](31-native-fastembed-semantic-code-search.md) is retired by - rejected decision 11 and is archival. + Dense retrieval is retired by rejected decision 11. Indexing runtime is `tracedecay-code-index-runtime`; generation retention is `tracedecay-code-index-retention`. - Policy and delivery surfaces: [Plans 06](06-policy-crate.md), [08](08-tool-catalog-crate.md), @@ -999,8 +994,7 @@ Unmeasured speculative optimizations and placeholder benchmarks do not ship. [Plan 39](39-embedded-grafeo-graph-database.md)'s embedded graph storage cutover and [Plan 40](40-partitioned-generation-storage-and-zero-copy-replay.md)'s - partitioned generation storage, which supersedes the monolithic sealed - generation envelope as the long-term storage design. + partitioned generation storage. Embedded graph storage ([Plan 39](39-embedded-grafeo-graph-database.md)) threads through the active @@ -1031,10 +1025,9 @@ debris quarantine, and one-content-copy enforcement. Session retention keeps raw and projected evidence deduplicated while LCM summary nodes retain losslessly expandable source references. Plan 09 exposes the corresponding storage findings through read-only Doctor composition over Plan 26 telemetry; -Doctor never invokes cleanup, GC, retention, relink, or recovery. Store soft -budgets are owner-configured and inert by default. Direct validation uses an -isolated profile and proves size reduction without reading or mutating the -operator profile. +Doctor never invokes cleanup, GC, retention, relink, or recovery. Direct +validation uses an isolated profile and proves size reduction without reading +or mutating the operator profile. The remaining delivery proceeds by complete product journeys under this index. Branch, pull-request, merge, worktree, and SHA choreography is not roadmap diff --git a/docs/plans/tracedecay-v2/05-query-crate.md b/docs/plans/tracedecay-v2/05-query-crate.md index a563a0af21..3c4a0fcf86 100644 --- a/docs/plans/tracedecay-v2/05-query-crate.md +++ b/docs/plans/tracedecay-v2/05-query-crate.md @@ -22,15 +22,14 @@ same-host measurement showing a smaller frequently touched compile graph. - The memory delivery adds facts and provenance, the temporal-kernel delivery adds LCM/session retrieval, the code-intelligence delivery adds - lexical code search, code navigation, and current diagnostics, and the - semantic delivery adds semantic search. + lexical code search, code navigation, and current diagnostics. - The application delivery composes query use cases in application and policy. The surface delivery exposes them through CLI, MCP, HTTP, LSP, export, and live non-dashboard adapters. The dashboard delivery first ships dashboard binding and dashboard parity. - Plan 05 owns the shared execution semantics. Plan 23 remains the current - owner of temporal retrieval semantics; Plans 25 and 31 own active - lexical/code and semantic acceptance. Plans 09/10/11/12/14 and Plan 24 + owner of temporal retrieval semantics; Plan 25 owns active lexical/code + acceptance. Plans 09/10/11/12/14 and Plan 24 consume the accepted temporal-kernel behavior, not a superseded implementation shape. - The task/work delivery journey reuses these shared scope, budget, cancellation, cursor, watermark, merge, and coverage primitives to execute diff --git a/docs/plans/tracedecay-v2/15-search-quality-evaluation-and-retrieval-research.md b/docs/plans/tracedecay-v2/15-search-quality-evaluation-and-retrieval-research.md index 58ab928292..dcfdb4df2c 100644 --- a/docs/plans/tracedecay-v2/15-search-quality-evaluation-and-retrieval-research.md +++ b/docs/plans/tracedecay-v2/15-search-quality-evaluation-and-retrieval-research.md @@ -20,7 +20,7 @@ graph, Plan 13 diagnostic anchors, or their authorization rules. Plan 15 owns retrieval quality, composition, evaluation, and profile-selection rules. Plan 25 owns code retrieval and shared-code facts. The application, dashboard, task/work, and public surface plans consume tested behavior and -evaluation results. Plan 31 is archival. +evaluation results. ## Outcome diff --git a/docs/plans/tracedecay-v2/16-cross-project-repository-worktree-scope.md b/docs/plans/tracedecay-v2/16-cross-project-repository-worktree-scope.md index 12ffb77a40..de39953e69 100644 --- a/docs/plans/tracedecay-v2/16-cross-project-repository-worktree-scope.md +++ b/docs/plans/tracedecay-v2/16-cross-project-repository-worktree-scope.md @@ -49,8 +49,8 @@ right data. ## Worktree index identity and reuse -Plan 16 composes the incremental index delivered by Plans 25/31; it does not -create a second watcher, parser, chunker, vector store, or scheduler. +Plan 16 composes the incremental index delivered by Plan 25; it does not +create a second watcher, parser, chunker, or scheduler. - Every linked worktree has an exact typed identity, fenced snapshot frontier, immutable code generation, freshness, and coverage. A shared Git common diff --git a/docs/plans/tracedecay-v2/25-code-intelligence-indexing-crate.md b/docs/plans/tracedecay-v2/25-code-intelligence-indexing-crate.md index 0fc40d61ea..d4f988dfac 100644 --- a/docs/plans/tracedecay-v2/25-code-intelligence-indexing-crate.md +++ b/docs/plans/tracedecay-v2/25-code-intelligence-indexing-crate.md @@ -75,7 +75,7 @@ regresses.) Plan 25 owns code-generation, chunking, graph, and generation-bound evidence semantics, including source-bound shared-code facts sealed into each generation. Plan 15 owns quality evaluation. Application, transport, and -dashboard plans consume the tested operations later. Plan 31 is archival. +dashboard plans consume the tested operations later. ## Outcome diff --git a/docs/plans/tracedecay-v2/26-observability-accounting-and-usage.md b/docs/plans/tracedecay-v2/26-observability-accounting-and-usage.md index 42424860fc..7acdf3d377 100644 --- a/docs/plans/tracedecay-v2/26-observability-accounting-and-usage.md +++ b/docs/plans/tracedecay-v2/26-observability-accounting-and-usage.md @@ -145,7 +145,7 @@ vocabulary as current ownership. cancellation; analyzer startup, restart, and indexing/degraded state; cache reuse and overlay freshness; diagnostic add and clear; provider conflicts; host delivery path; partial coverage and drops; and bridge reconnect. -- Plans 25/31 emit bounded incremental-index observations for exact worktree +- Plan 25 emits bounded incremental-index observations for exact worktree identity classes without paths: event-to-reconcile and event-to-ready latency; debounced/raw event counts; overflow/rescan reason; candidate, hashed, parsed, and no-op file counts; Tree-sitter changed-range bytes; diff --git a/docs/plans/tracedecay-v2/31-native-fastembed-semantic-code-search.md b/docs/plans/tracedecay-v2/31-native-fastembed-semantic-code-search.md deleted file mode 100644 index f4e0c99c15..0000000000 --- a/docs/plans/tracedecay-v2/31-native-fastembed-semantic-code-search.md +++ /dev/null @@ -1,708 +0,0 @@ -# Native FastEmbed semantic code search (archival) - -**Status:** superseded on 2026-09-13 by -[rejected decision 11](00-plan-set-index.md#rejected-approaches). This file -records the retired dense-retrieval design. It is not current implementation, -product, storage, or test authority. - -TraceDecay no longer downloads code-search models, loads an inference runtime, -builds vector generations, exposes vector search, or runs semantic activation, -qualification, calibration, reranking, or rollback. Plans 05, 15, and 25 own -the retained exact, lexical, graph, and source-bound shared-code behavior. - -The content below is historical. It must not be used to restore a fourth -semantic code-intelligence authority. - -## Outcome - -TraceDecay augments exact lexical and graph search with local code embeddings. -Exact results remain authoritative without a model; similarity alone never proves impact, lineage, or equivalence. - -## Ownership - -- Plan 25 builds deterministic, storage-neutral `CodeSearchDocumentV1`, - `CodeSearchChunkV1`, `ChangedCodeChunkSetV1`, and - `CodeIndexCapabilityManifestV1` values from an exact code snapshot. -- Plan 04 defines deterministic changed-input projection, checkpoint, retry, - and publication semantics. Daemon/service orchestration decides when that - resumable work is scheduled. -- Plan 02 stores immutable vector generations, manifests, checkpoints, the - atomic active-generation pointer, and chunk projection receipts through - daemon-owned writer authority. -- Plans 05/15 own the common `CompactCandidate`, independent lane, deterministic - fusion, contribution, diversity, rerank, hydration, cursor, fallback, - explanation, and redundancy contracts shipped by federated retrieval. This plan adds one semantic - adapter; it does not create a parallel code-search kernel. -- Plan 20 owns versioned model/reranker profile selection, configuration - precedence, and atomic active/rollback profile pointers. This plan owns the - root-private versioned artifact manifest and SHA-256 integrity verifier, - FastEmbed runtime adapter, - bounded sessions, semantic generation service, and their typed ports. The - Plan 09 application layer later composes those accepted ports; it does not - retroactively own this plan's model implementation. -- Daemon/application orchestration schedules changed eligible chunks and - rebuilds. Plan 04 owns deterministic projection, checkpoint, retry, and - publication semantics, not scheduling policy. -- Plan 15 exclusively owns retrieval-research design, versioned corpora and - labels, metrics and strata, candidate profile comparison, thresholds, and - activation recommendations. This plan implements measured profiles and emits - results; it cannot tune or activate itself from aggregate or public benchmark - rank. -- Plans 10/11/20/21 expose the same application operations through API, - dashboard, configuration, CLI, and MCP. - -Only one root-private adapter depends on `fastembed`. Crates for indexing, store, -query, API, and UI depend on ports and stable domain values, never -FastEmbed runtime types. - -## Behavioral contracts and historical layout - -The semantic boundary must preserve the ownership and fields below. The listed -paths and Rust names are a historical design sketch, not artifact-name parity -requirements. Current owner-approved names and locations are valid when direct -contract regressions prove the same lane isolation, generation compatibility, -authorization, coverage, fallback, and receipt behavior. - -The original semantic-delivery design proposed these files: - -- `crates/tracedecay-domain/src/code_intelligence/search.rs`: - `EmbeddingProjectionKeyV1`, `SemanticSearchIndexKeyV1`, - `SemanticCapabilityManifestV1` and - code-specific semantic evidence values. Generic ranked-list, compact- - candidate, contribution, fusion, rerank, hydration, and cursor values remain - in federated retrieval's `crates/tracedecay-domain/src/retrieval.rs`. - Semantic projection uses Plan 25's `CodeChunkProjectionReceiptV1` and - `ProjectionBatchReceiptV1`; no semantic-specific receipt authority exists. -- `src/semantic_code/artifacts.rs`: versioned manifest parsing, SHA-256 member - integrity verification, import staging, atomic install, inventory, retention, - and quarantine. -- `src/semantic_code/manifest.rs`: model/runtime/projection and capability - manifest validation. -- `src/semantic_code/projector.rs`: changed-chunk batching, receipt production, - resumable checkpoints, and publication handoff. -- `src/semantic_code/fastembed_adapter.rs`: the only `fastembed` import and the - only model inference implementation. -- `src/semantic_code/session_pool.rs`: bounded sessions keyed by the complete - projection/privacy identity. -- `src/query/retrieval/semantic.rs`: the code-semantic implementation of the - federated `Retriever` port. It emits generic `CompactCandidate` values carrying a - typed code anchor and semantic evidence; it consumes neither lexical nor - graph candidates. -- `src/semantic_code/service.rs`: temporary root-private semantic orchestration for - artifact/runtime admission, projection generation, shadow execution, and - activation/rollback. It exposes typed ports consumed by the application - layer and contains no transport binding. -- `src/config/retrieval.rs`: Plan-20-owned versioned profile definitions and atomic - active/rollback profile pointers. -- `src/store/vector_generations.rs`: Plan-02-owned implementation of the - semantic projection read/write ports. Query and semantic modules do not - depend on its physical schema. - -The semantic adapter extends the federated retrieval contract rather than replacing -it: - -```rust -pub struct SemanticRetrievalRequestV1 { - pub query_digest: QueryDigest, - pub query_view: EphemeralSanitizedQueryViewV1, - pub authorized_scope: AuthorizedScopeReceiptV1, - pub authorization_epoch: AuthorizationEpoch, - pub scope_digest: ScopeDigest, - pub code_generation: CodeGenerationId, - pub projection_key: EmbeddingProjectionKeyV1, - pub capability_manifest_digest: ManifestDigest, - pub budget: RetrievalBudget, - pub continuation: Option, -} - -pub struct SemanticCodeRetriever { - // Root-private FastEmbed session and semantic projection read ports. -} - -// SemanticCodeRetriever implements -// Retriever. - -pub struct CodeSemanticEvidenceV1 { - pub projection_key: EmbeddingProjectionKeyV1, - pub vector_generation: VectorGenerationId, - pub chunk_id: CodeSearchChunkId, - pub distance: CanonicalSemanticDistanceV1, - pub search_kind: SemanticSearchKindV1, -} -``` - -The adapter implements the one federated `Retriever` port and receives one -frozen authorized scope, code generation, semantic -projection, and bounded ephemeral sanitized query view. It returns the generic -Plan 15 batch with one `CompactCandidate` and one occurrence-keyed -`CodeSemanticEvidenceV1` per result. It cannot mint exact -admission, read another lane's candidates, fuse, diversify, rerank, hydrate, or -retain raw query text/query vectors. Store conformance uses separate lexical, -graph, vector, receipt, and hydration representations; no table, vector index, -or materialized join becomes authority. -Deadline and cancellation limits are fields of the shared `RetrievalBudget`; -this plan does not introduce semantic-only budget or continuation types. - -## Deterministic documents and generations - -Each chunk records repository/project/worktree/ref/snapshot identity, immutable -code generation, file and symbol identity, source span, language/extractor and -chunker versions, sensitivity decision, content digest, stable ordinal, and -bounded sanitized text. Symbol boundaries are preferred; oversized symbols use -versioned structural splits. Generated, vendor, binary, ignored, fixture, and -unsupported content has an explicit classification. - -A vector-generation identity includes the ordered eligible-document manifest, -model/tokenizer/runtime manifest, dimension, metric, normalization, chunker, -privacy domain/key epoch, and source watermark. The manifest also pins query -and document instructions/prefixes, pooling, truncation side and length, -precision/quantization, runtime/backend/thread/device identity, and search or -ANN parameters when present. Builds checkpoint in bounded batches. Partial or -mixed generations are never queryable. Publication verifies membership, -dimensions, finite values, digests, and watermark before one atomic pointer -swap; deletion creates a tombstone and unchanged inputs do no embedding work. - -`EmbeddingProjectionKeyV1` pins every vector-affecting input: model artifact, -tokenizer/config and instruction digests, pooling, truncation side/length, -runtime/backend/build revision, deterministic device class, dimension, metric, -normalization, precision/quantization, chunk schema/chunker revision, privacy -domain, and key epoch. Its canonical digest is Plan 25's -`ProjectionKeyV1.profile_digest`. Search/ANN structure parameters use a -separate `SemanticSearchIndexKeyV1`; changing them rebuilds only derived search -structures and query caches. Thread/batch settings remain execution-manifest -evidence and must produce the same vector digest or fail publication. Vectors -and projection rows are replayable derived data; canonical chunks and exact -code-generation evidence remain authority. - -For each `ChangedCodeChunkSetV1`, the projector returns one ordered -`CodeChunkProjectionReceiptV1` per added/changed/deleted chunk and an explicit -aggregate reused count in `ProjectionBatchReceiptV1`. A receipt records source -generation and manifest, projection key, chunk ID, prior/current digest, -`embed | tombstone`, completed/skipped/failed outcome, and output digest. -Deleted receipts bind `prior_generation`/`prior_chunk_digest`, the batch -`request_digest`, and have no current digest. -Reused chunks produce no per-chunk receipt. A no-op request with an unchanged -target projection key invokes no inference; a key-only replay expands reused -eligible chunks into explicit embed operations before dispatch. A one- -symbol edit embeds only its changed chunks and affected file-level chunks. A -model, tokenizer, instructions, runtime-compatibility, dimension, metric, -normalization, precision, privacy-domain, or key-epoch change creates a new -projection generation by replaying retained canonical chunks without reparsing -unchanged source. Old retained projections remain immutable and addressable by -their exact key; partial receipt sets never activate. - -### Worktree-aware projection scheduling - -- Consume Plan 25's reconciled `ChangedCodeChunkSetV1`; do not watch files, - rescan repositories, invoke Git status, or infer changes inside the semantic - runtime. -- Batch only added/changed eligible chunks through FastEmbed - `TextEmbedding::embed`. Construct the model through - `try_new_from_user_defined` from explicitly installed local ONNX/tokenizer/ - config bytes. Query and projection paths never invoke the hub, discover an - ambient model cache, or download artifacts. -- A no-op changed set performs zero model calls. Deletions produce tombstones - without inference. A projection-key change replays retained eligible chunks - without parser/extractor work; a search-index-key-only change rebuilds only - the derived exact-flat/search structure. -- Reuse vector bytes across worktrees only when canonical chunk content, - projection key, privacy domain, and key epoch match. Every projection row, - receipt, and immutable vector generation remains bound to its exact source - worktree snapshot and code generation. The committed retrieval profile names - that vector generation, while each admitted query separately pins the exact - source code generation; neither identity is inferred from a process cache. -- Coalesce superseded projection batches by exact worktree and source - generation. Bound queued bytes, batch size, session count, model memory, and - publication concurrency. Interactive exact/lexical/graph queries have - priority; semantic query admission is fail-fast and never enters the - projection waiter queue. -- Keep a prior semantic generation queryable only when its complete projection - key and source compatibility still match the frozen code request. Otherwise - omit semantics until the new complete generation atomically publishes. - -## Model and offline lifecycle - -Configuration selects a versioned embedding profile and, independently, an -optional reranker profile. With automatic acquisition enabled, the daemon -queues the selected catalog model's immutable Hugging Face repository revision -in the background. Manifests pin actual model/tokenizer/config bytes, licenses, -runtime/build identity, dimensions, normalization, metric, device, threads, -batching, and resource ceilings. Implementation selects maintained -crate/runtime versions during the semantic delivery; activated model and reranker profiles still -require a passing Plan 15 evaluation. This plan contains no stale crate or -model-version pin. - -Every acquisition/import path verifies artifacts before installation or -activation. Queries never download a model or open an ambient cache. Offline -startup remains healthy and baseline-complete. Compatible warmed sessions -are pooled under bounded memory, concurrency, idle, and cancellation policy. -Load failure, OOM, corruption, missing bytes, or incompatible pins disables the -affected semantic stage without silently selecting another model. - -Artifact handling is explicit: - -- A versioned canonical manifest lists profile kind, model/tokenizer/config file digests, - byte lengths, licenses, upstream source metadata, runtime compatibility, - projection inputs, and the complete resource ceiling. The manifest and each - declared member are identified by SHA-256 solely to detect drift, corruption, - or incomplete installation. These versioned manifests and integrity checks - are the complete artifact-verification contract. -- Daemon-owned background acquisition resolves only the cataloged repository - and immutable revision into an explicit cache under the lifecycle root. It - stages each member, independently checks its declared length and SHA-256, - and atomically publishes the complete verified install. Query/runtime paths - perform no download, import, extraction, source discovery, or network - inference. -- Explicit local-path and configured-HTTPS import APIs remain available as - manual operations, but no production surface currently wires them. They are - not the delivered acquisition journey. -- Interrupted imports are resumable only when the source supplies immutable - length, digest, and range identity; otherwise staging is discarded. Archive - traversal, symlink/hardlink entries, absolute paths, duplicate members, - undeclared members, size expansion beyond the manifest, and digest mismatch - quarantine the import without exposing it to runtime discovery. -- Installed artifacts live in one Plan-02-owned user store keyed by versioned - manifest digest, never an ambient Hugging Face/ORT/FastEmbed cache. The - daemon's lifecycle-root Hugging Face cache is an acquisition source only, - not runtime or installation authority. Inventory records distinguish - `staged | verified | installed | quarantined | retained_for_rollback`. - Garbage collection may remove only unreferenced non-active/non-rollback - artifacts after a daemon lease and append-only receipt. -- Activation is an authenticated Plan 20 compare-and-swap from the expected - active/rollback profile digests. It verifies artifact inventory, accepted - Plan 15 report, runtime/platform compatibility, complete projection - generation, resource ceiling, and executable rollback before publishing. - Rollback uses the same checks and must pass a cold-start/offline drill. - -The mandatory Plan 25 `CodeIndexCapabilityManifestV1` admits lexical/graph -retrieval. Optional `SemanticCapabilityManifestV1` pins the authorized scope -digest, code and vector generations, projection and search-index keys, -supported chunk grains/languages, coverage, partial states, privacy domain/key -epoch, and manifest digest. Fusion, candidate budgets, diversity, reranking, -and hydration are validated separately from the active Plan 15/20 profile. The semantic service validates the base -manifest before any channel and validates the semantic augmentation only before -semantic/rerank work. Missing semantic capability yields lexical/graph mode; an -explicit strict-semantic request yields the typed unavailable result. - -## Query and redundancy - -Search resolves exact scope and frozen generation first, reproduces the -versioned exact+lexical+graph baseline, then adds a compatible semantic -candidate list. Fusion is stable and explainable; Plan 15's exact-admission -authority keeps exact identifiers, paths, quoted phrases, errors, tool names, -and configuration keys in a non-demotable tier. The first production semantic -baseline is deterministic exact flat-vector search unless measured -current/10x evidence shows it violates a reviewed resource budget. Optional -reranking is bounded to a configured top-N candidate set, is admitted only -after candidate-controlled gain with no protected-stratum regression, and -preserves the pre-rerank list byte-for-byte when unavailable. Raw similarity, -logits, margins, or fused scores are not confidence; calibrated abstention -requires a versioned cohort/generation-bound profile and reports invalid or -shifted calibration explicitly. Strict semantic requests return a typed -unavailable result. - -Semantic projection and indexing run asynchronously. Existing exact, lexical, -and graph operations remain callable and return without joining, polling, or -waiting for semantic work. The semantic lane is omitted until one complete -compatible immutable vector generation becomes current through a single atomic -publication. Staged, partial, indexing, stale, failed, cancelled, or -incompatible generations contribute no candidates, score, cap pressure, -cursor bytes, or rank effects. A request for strict semantic alone may return -typed unavailable; it may not delay or replace the normal federated path. - -`code.redundancy` reuses the same active generation. It canonicalizes pairs, -removes self/overlapping chunks, and reports `exact_clone`, -`structural_near_duplicate`, `semantic_analogue`, or `insufficient_evidence`. -Semantic-only matches remain review candidates, never automatic rewrites or CI -violations. Disabled semantics preserves the structural baseline and ordering. - -The Plan-05 query pipeline executes these phases without combining them: - -1. Resolve and authorize project/repository/worktree/ref scope; freeze code, - exact, lexical, graph, and compatible semantic generations; validate the - base and semantic capability manifests, authorization epochs, and budgets. -2. Reuse federated retrieval's separately inspectable `RetrieverOutcome` values for - `ExactLiteral`, `Lexical`, and `Graph`; produce one independent - `Semantic` outcome. No semantic code calls, wraps, filters, or mutates - another lane. -3. Validate/canonicalize the semantic outcome under the generic Plan 15 lane - contract: finite canonical scores, candidate identity dedupe, generation and - authorization equality, stable lane-local ordering, complete coverage, and - a bounded continuation. -4. Pass all compact candidates to the one federated fusion implementation. The - existing exact tier remains lexicographically first and cannot be demoted by - graph/semantic scores, diversity, or reranking. This plan evaluates semantic - weighting/fusion candidates, including exact rational RRF if proposed, on - the same recorded candidate inputs; no algorithm, constant, or weight - activates without a passing Plan 15 evaluation. -5. Use the complete comparator and diversity policy from the tested generic - profile. The cursor binds emitted identities, ordered overflow, every lane - continuation, profile/generation/projection digests, and authorization - epoch. Input insertion order, hash order, and completion order cannot affect - IDs, order, contributions, explanations, coverage, or cursor bytes. -6. Retain every generic `CandidateContribution`, including lane-local raw score - domain/rank, calibrated fixed-point feature when valid, weighted - contribution, profile/projection identity, evidence class, and exclusion - reason. Graph path coverage/authority and semantic projection/distance - provenance remain typed lane evidence. A fused score is not probability or - confidence. -7. Optionally rerank only the profile-bounded non-protected candidate set while - preserving the pre-rerank list and contributions. Absence/failure returns - those bytes unchanged. -8. After fusion, diversity, reranking, and page selection, reauthorize and - hydrate only selected hits through the federated hydration port. Hydration may add bounded - symbol text, file context, declarations, and graph neighbors from the same - generation; it cannot add candidates or change rank. - -Plan 15's generic `FusionProfile` pins the selected algorithm, lane features/ -weights, candidate budgets, diversity caps, protected exact classes, rerank -budget, hydration budget, and total comparator. Plan 15 recommends that profile -from direct evaluation; Plan 05 implements it, and Plan 31 supplies -only the semantic lane, artifact/runtime/projection implementation, and -measurements. - -## Authorization and local/private boundary - -- Owning source stores authorize exact scope, operation, privacy domain, key - epoch, and code generation. Plan 20's accepted profile and the semantic service - separately authorize semantic projection generation and runtime profile. The - semantic query composition validates both receipts before invoking the semantic - lane; source denial invokes zero channels, while - semantic denial invokes no semantic/rerank port and preserves authorized - lexical/graph execution. -- Candidate references carry authorization receipt ID and epoch without private - payload. Immediately before hydration, the shared query pipeline rechecks current - authorization, scope, privacy domain/key epoch, and frozen generation. - The owning store performs atomic receipt validation plus bounded payload/ - neighbor read; every neighbor must be inside the authorized scope. Revocation - before or during a multi-read discards the complete hydrated hit and returns - a typed denial with no source text or neighbor payload. -- All caches are local and privacy-domain/key-epoch separated. Model artifacts - key by versioned manifest digest; sessions by embedding projection key; vectors - and checkpoints by projection key plus source generation/chunk digest. - Result caches additionally key by authorization receipt/epoch, authorized - scope and query digests, fusion/rerank/request revisions, and pagination. - Hydration caches additionally key by candidate identity, requested fields, - and neighbor budget. Domain/epoch changes produce zero hits in every cache; - authorization changes produce zero result/hydration hits. No ambient model - cache, network inference, browser runtime, external process, or cross-domain - cache is permitted. -- Raw queries, source text, vectors, private paths/symbols, and hydrated - explanations never enter telemetry or checked-in benchmark artifacts. - A raw query is sanitized into a bounded ephemeral query view before model - inference. `EphemeralSanitizedQueryViewV1` contains bounded sanitized bytes - plus sanitizer and normalization revisions; it is non-serializable, - non-cacheable, and valid only for the authorized request. `QueryDigest` is a - privacy-domain/key-epoch keyed MAC over that view, never an unkeyed content - hash. `QueryDigest` is owned by - `crates/tracedecay-domain/src/retrieval.rs` and may appear only in in-process - request state, authenticated cursor identity, and privacy-separated local - cache keys. `EphemeralSanitizedQueryViewV1` is owned by - `code_intelligence/search.rs`. Neither value may enter receipts, telemetry, - durable stores, or checked-in artifacts. Query text and vectors remain in - memory only for that request. - Operational receipts contain opaque identities, revisions, outcomes, counts, - and digests only. Only sanitized fixtures and aggregate Plan 15 reports may - enter Git. -- Semantic errors, timeout, cancellation, OOM, corruption, or missing artifacts cannot - broaden scope. When the selected profile permits fallback, the lexical/graph - lane outcomes and Plan 15's named exact/lexical/graph fallback subpayload are byte-identical. - The enclosing response may add only a typed semantic/rerank outcome outside - that subpayload, its digest, and cursor identity; strict mode fails closed. - -## Plan 15 evaluation handoff and fresh-store reset - -Semantic retrieval ships a versioned sanitized corpus covering exact names, natural-language -intent, mixed queries, renamed symbols, same-name cross-scope cases, no-answer -queries, generated/vendor noise, large symbols, unsupported languages, and -incremental edits. It measures exact-hit retention, precision/recall/MRR/nDCG, -wrong-scope and no-answer error, worst strata, build/update time, p50/p95/p99, -CPU/RSS, model/vector/cache bytes, cancellation, and offline behavior. - -Plan 15 owns this corpus's partition/label policy, metrics, thresholds, and -activation recommendation. This plan owns reproducible Linux execution and truthful -`pass`, `fail`, or `pending` summaries. Activation requires a passing Plan 15 -evaluation showing no scope/privacy or protected -exact/no-answer/wrong-scope regression, demonstrated semantic gain, and -declared current/10x Linux resource observations. Sensitive or -ineligible bytes never enter documents, artifacts, metrics, explanations, or -model-assisted routes. - -Late interaction, quantization, and specialized ANN remain measured candidate -profiles, not semantic-delivery defaults. ANN is admitted only when it beats exact search's -reviewed resource budget while meeting exact-oracle average, tail, minimum, -and zero-recall-query gates under immutable-generation compatibility. No HNSW, -DiskANN, ScaNN, vector database, precision, or quantization choice is -mandatory. Public benchmark rank cannot select a production profile. - -Non-final vectors are never trusted, republished, or converted. They return -`ResetRequired`; an operator explicitly resets or recreates the generation from -current canonical documents without using the old vector as input. - -## Current delivery audit - -The audit below separates callable delivery from pending direct execution. A -present source file or type name is not completion by itself; the cited direct -regressions invoke the relevant boundary. Static fixture validation is useful, -but the current Linux evaluation remains pending. - -**Production lifecycle correction (updated 2026-07-26).** Commit `dd4adbe2a` -ships daemon-owned immutable `hf-hub` background acquisition. -`shared_lifecycle_owner()` opens a lifecycle-root-scoped -`HfHubModelMemberSourceV1`; startup queues acquisition when configured, and the -source resolves only the cataloged repository/revision before the lifecycle -checks every member's declared length and SHA-256 and atomically installs the -package. Status and Doctor expose downloading, verifying, installed, and -failed states. A clean temporary profile downloaded and digest-verified the -full default Jina model end to end, and packaged offline acceptance installs -the same verified model from the daemon-owned cache. The explicit local and -configured-HTTPS import APIs remain manual and production-unwired; they are not -being certified by this correction. Installed model bytes remain semantically -omitted until compatible vector indexing publishes readiness, preserving the -asynchronous non-blocking contract. - -**Why the old green gate was misleading.** Before `dd4adbe2a`, -`HfHubModelMemberSourceV1::fetch_member` unconditionally returned -`semantic model lifecycle operation rejected`, so no shipped binary could -acquire a model. The distribution gate's packaged-semantic half tested only -typed fallback/strict-unavailable behavior, the exact terminal absence produced -by that rejection, and therefore encoded the bug as its passing specification. -The corrected gate now runs packaged background acquisition and verifies the -installed Jina members; restoring the unconditional rejection makes that gate -fail, and restoring the source makes it pass. - -- **Library-first FastEmbed:** delivered. The root manifest keeps - `fastembed` optional with upstream defaults disabled, the - `semantic-fastembed` feature selects the native runtime, and - `FastEmbedEmbeddingRuntime::open_session` uses FastEmbed's user-defined local - byte constructor. Native model execution and resource evidence remain - pending. -- **Default production posture:** delivered in the root feature manifest. - FastEmbed/ORT is available on Linux and macOS. Windows production builds keep - the daemon, hosts, exact/lexical/graph retrieval, and the ORT-free Model2Vec - backend; selecting a FastEmbed model reports the typed unavailable runtime - state and never silently substitutes another embedding backend. -- **Daemon-owned immutable acquisition:** delivered. Startup queues the - selected catalog revision without blocking startup or query paths; the - lifecycle-root source cache is explicit, and verification plus atomic install - remain independent of `hf-hub`. -- **Local verified model bytes only:** delivered at the runtime boundary. - Model, tokenizer, and config bytes come from the installed manifest members; - runtime construction has no hub, ambient-cache, download, external-process, - or network-inference path. Background acquisition now creates the verified - installed manifest and members; explicit local/HTTPS imports remain - production-unwired. Direct tests validate constructor and integrity reads, - while cross-platform runtime coverage remains normal CI work. -- **Exact-flat semantic baseline:** delivered by - `SemanticCodeRetriever`/`SemanticVectorReadPort::scan_exact_flat`, with direct - deterministic ordering, provenance, generation, and coverage regression in - `exact_flat_scan_is_deterministic_and_emits_generic_semantic_evidence`. -- **Immutable generations and verified publication:** active integration work - uses the staged vector-generation store and the Plan 39 verified Grafeo - publication boundary. Staged work is not queryable, cancellation preserves - every prior verified generation, and failed publication cannot expose a - partial generation. Publication only adds immutable catalog state and may - advance the relational verified graph head; it never activates semantic - routing. Plan 20's linked configuration transition and activation receipt - are the sole active/rollback authority, while process-local runtime pointers - are observed readiness caches only. -- **Asynchronous, non-blocking indexing:** delivered at the projection, - generation-selection, runtime-routing, and bounded-session boundaries. - `only_a_current_receipt_routes_to_semantic_search` routes indexing, degraded, - unavailable, and rollback states to the frozen exact/lexical/graph fallback, while - `saturated_runtime_omits_semantics_without_entering_the_waiter_queue` proves - query work does not wait for semantic capacity. -- **Calibrated abstention:** delivered. Missing or shifted calibration invokes - no semantic authority and preserves fallback; distance and margin rejection - are versioned and generation-bound. Strict semantic returns typed unavailable. - Linux threshold evaluation remains pending. -- **Byte-stable exact/lexical/graph fallback:** delivered at the semantic service boundary. - Direct regressions retain the caller-owned validated fallback object through - augmentation and every tested abstention path. The versioned fallback bytes - and digest remain pending direct lexical verification. -- **Evaluation before activation:** still pending. Current runtime - routing requires an observed current activation receipt and rejects an - indexing receipt, but the Plan 15 Linux evaluation, current/10x measurements, - normal cross-platform CI, and rollback drill are still incomplete. - `result-pending.json` therefore keeps activation false and truthfully reports - `pending`. - (Update 2026-08-07: the named artifact is stale, no `result-pending.json` - exists in-tree (`git ls-files` returns zero hits). The behavior it stood for - is intact and is now asserted by the evaluation harness itself: - `tests/search_eval_cli_test.rs:53` - (`compare_reports_unmeasured_semantic_and_rerank_stages_as_pending`) requires - the `compare` CLI to exit non-zero and report unmeasured semantic/rerank - stages as `pending`. The gate is unchanged; only the file reference is - obsolete.) - -## Planned behavioral delivery and direct verification - -The native semantic delivery remains unfinished. The checkpoints below are required product behavior. -Paths, type spellings, fixture filenames, benchmark entrypoints, and test-suite -registration are non-normative historical suggestions. Completion follows the -callable semantic operation, direct regressions, the Plan 15 Linux evaluation, -and normal CI. - -1. **Contracts and capability admission:** add semantic-only domain values, - ports, manifest validators, ephemeral query-view rules, and split in-memory - adapters. Reuse federated retrieval's generic retrieval/fusion/hydration behavior. Direct - regressions cover contract validation, capability admission, and storage - independence. -2. **Artifact and runtime foundation:** implement one root-private artifact - verifier, manifest validator, FastEmbed adapter, and bounded session pool. - Direct regressions cover versioned manifest and SHA-256 member verification, - local and explicit HTTPS import, - traversal/expansion rejection, interrupted staging, atomic install, - quarantine/GC, cold and warm sessions, OOM, cancellation, - offline startup, no ambient cache, and Linux/Windows native-runtime - compatibility. -3. **Incremental vector projection:** implement the projector and vector-generation - store against Plan 04 projection/checkpoint semantics, Plan 02 - receipt/publication authority, and this plan's runtime ports. Daemon/service - orchestration owns asynchronous worktree-fair bounded scheduling; do not - assign it to Plan 04. Direct regressions cover changed-chunk batching, - deletion without inference, no-op zero inference, cross-worktree physical - reuse without identity reuse, superseded-batch cancellation, model-key - replay, search while indexing without waiting, omission of incomplete/ - incompatible generations, atomic publication, cancellation/failure - isolation, and offline lifecycle. -4. **Exact-flat semantic retrieval and shadow composition:** implement one - independent semantic outcome and compose it with frozen federated lane outputs - through the shared fusion/diversity/cursor behavior. Direct regressions - cover lane isolation, contribution provenance, protected exact results, - diversity/pagination, and byte-identical fallback. -5. **Late hydration, privacy, and rollback:** reuse federated hydration with semantic - profile admission, generation checks, authorization recheck, - domain-keyed caches, payload-safe receipts, active/rollback pointer CAS, and - cold offline rollback. Direct regressions cover hydration, authorization, - privacy-domain isolation, activation, and rollback. -6. **Developer evaluation:** use reproducible checked-in Linux workloads, - explicit reset/recreation checks, exact-flat oracle comparisons, focused - channel/fusion/calibration/rerank ablations, privacy/non-interference checks, - and current/10x resource observations. Optional ANN and reranker branches - begin only after the exact-flat shadow checkpoint and cannot enter the - critical path without a passing comparison. -7. **Activation and verification:** only a passing Plan 15 result may make - semantics eligible for the existing configuration activation. Run staged - shadow/cohort behavior, rollback, fresh-store reset, privacy, architecture, direct - tests, Linux evaluation, and normal all-feature cross-platform CI. Status - and Doctor behavior are exercised through the production lifecycle states; - activation still requires the remaining evaluation and indexing evidence. - This plan does not create a temporary public semantic endpoint or reserve later - surface contracts. - -A reproducible projection workload measures clean, warm one-symbol, deletion, -no-op, model-key replay, cancellation, and incompatible rebuild. A query -workload reports lexical, graph, semantic, fusion, rerank, and hydration time -separately at current and 10x corpus sizes, -including p50/p95/p99, CPU, peak RSS, model/vector/cache bytes, candidates per -channel, chunks embedded/reused/deleted, hydration fetch count, and fallback. -Channel ablations use equal candidate budgets; exact flat-vector search is the -semantic oracle. End-to-end performance work consumes these production -measurements, while Plan 15 owns quality/resource trade-off and activation -recommendations. - -The query workload also overlaps queries with a blocked projection worker and -with staged, partial, failed, cancelled, stale, and incompatible generations. -It records exact/lexical/graph completion independently, requires zero wait on -the semantic worker, compares the visible federated fallback bytes and rank before -and during indexing, and observes semantic candidates only after the complete -compatible generation is verified and Plan 20 has committed the matching -activation receipt. Those are ordered publication and configuration -transactions, not a graph-local pointer swap. - -Each workload manifest pins corpus/query digests, exact file/chunk/query counts, -language/source strata, seed, model/projection/fusion revisions, hardware and -runtime summary, cache state, and concurrency. The 10x workload contains -exactly ten times the eligible chunks of the current workload without copying -quality labels across partitions. Reports retain the raw Linux samples needed -for every statistic and label unsupported p99 or uncertainty claims `pending`. - -### Hard activation barrier - -No semantic comparison starts until callable exact/lexical/graph behavior -passes direct regressions and its versioned profile, exact-tier contract, -fallback-subpayload bytes, and quality fixtures are identified by real -content/revision digests. Historical artifact-name parity is not part of this -barrier. No activation occurs unless Plan 15 reports `pass`, -authorization/scope leakage is zero, protected exact results are -unchanged, the exact/lexical/graph fallback subpayload is byte-identical, -generation compatibility holds, search-during-indexing leaves fallback bytes and rank -unchanged, incomplete or stale generations contribute nothing, all -resource ceilings pass, and cold offline rollback succeeds. - -A `fail` or `pending` result leaves semantics disabled and no semantic profile -eligible. - -## Acceptance - -Semantic retrieval is complete when projection, atomic publication, lexical-fallback-preserving -search, bounded generic fusion/reranking/redundancy, artifact/offline lifecycle, -configuration, production status/Doctor behavior, corpus/resource/privacy -tests, fault recovery, rollback, explicit reset/recreation, the Linux developer -evaluation, and normal CI pass. Application, public transport, and dashboard -adapters remain required parts of the same product. No separate -semantic endpoint, vector database, browser inference runtime, or model-specific -transport is introduced. Queries never silently substitute a model/revision, -download at query time, cascade to an unmeasured representation, or treat -semantic similarity as identity, impact, lineage, or equivalence. - -- A no-op chunk manifest performs zero embedding calls. A one-symbol edit - embeds exactly its changed symbol chunks and affected file-level chunks and - tombstones explicit deletions. Changing only vector-affecting model profile - fields with unchanged canonical chunk inputs replays all eligible chunks with - zero parser/extractor calls; chunker/sanitizer/sensitivity changes follow - Plan 25's canonical rebuild path. -- Receipt fixtures reject missing, duplicate, extra, wrong-generation, wrong- - digest, and wrong-key entries; crash/cancellation leaves the previous active - retrieval configuration unchanged and no partial projection queryable. - Queries issued while - projection is blocked complete through exact/lexical/graph without waiting - and match the versioned federated fallback bytes and rank. Only a complete, - recovered-state-verified generation named by the current linked activation - receipt may add the semantic lane. -- The semantic adapter emits a separately inspectable generic - `RetrieverOutcome>`; the exact, - lexical, and graph outcomes are unchanged. Disabling or failing semantic - retrieval preserves Plan 15's named federated fallback subpayload bytes. Fused - explanations reproduce every declared generic profile - contribution and the complete comparator for every result. -- Exact-hit retention and first-relevant-protected-hit Recall@10 are 100% over - Plan 15's versioned protected-query set, with numerator, denominator, and - support reported separately for symbols, qualified names, compiler/runtime - errors, CLI flags, paths, quoted phrases, tool names, and configuration keys. - For multi-target queries, every relevant protected hit admitted by the - requested page size precedes every fuzzy, graph-only, or semantic-only hit. -- One hundred runs with shuffled candidate insertion and channel completion - order produce byte-identical channel lists, fused IDs/order/contributions, - diversity spillback, explanations, and cursors. -- Fixture profiles with `max_non_exact_per_source = 3` and - `max_non_exact_per_file = 2` enforce those caps before pagination, preserve - protected exact hits, and deterministically refill from overflow candidates. - A three-page fixture proves no duplicate, omission, or cap drift after cursor - resume. -- A page requesting five hydrated hits reads payloads and neighbors for at most - those five hits and the profile's declared neighbor budget. Hydration cannot - change rank; every neighbor matches the frozen generation and preserves graph - ordered path identity, coverage, and weakest edge authority. -- Denied authorization invokes zero retrieval ports and zero payload reads. - Revocation before hydration returns no payload. Privacy-domain/key-epoch - changes yield zero session/vector/result/hydration cache hits, and cross- - domain candidates, vectors, receipts, metrics, or explanations are zero. -- Split-adapter conformance produces identical results when lexical postings, - graph evidence, vectors, receipts, and hydration payloads use separate stores, - proving embeddings and no single physical table are authority. -- Artifact fixtures reject malformed versioned manifests, undeclared or - duplicate members, path traversal, links, size expansion, - digest/length mismatch, incompatible runtime/platform pins, interrupted - publication, and deletion of active/rollback artifacts. Query execution - performs zero network/import/cache-discovery operations. -- Distribution acceptance starts from an isolated profile, queues the - background worker, installs every digest-verified default Jina member, and - proves semantics remain omitted until indexing readiness. Its offline mode - consumes only the lifecycle-root cache and must fail if production - acquisition regresses to unconditional rejection. -- The checked-in benchmark corpus contains only sanitized fixtures and expected - opaque anchors; raw private queries/source are absent. Activation requires a - passing Plan 15 evaluation and cannot be inferred from public rank or - aggregate gain. diff --git a/docs/plans/tracedecay-v2/38-storage-retention-size-and-efficiency.md b/docs/plans/tracedecay-v2/38-storage-retention-size-and-efficiency.md index c7f9a8191f..f53b66bc91 100644 --- a/docs/plans/tracedecay-v2/38-storage-retention-size-and-efficiency.md +++ b/docs/plans/tracedecay-v2/38-storage-retention-size-and-efficiency.md @@ -147,14 +147,9 @@ measurements, not inferred table sizes. ## Product contract -1. **Branch store lifecycle.** Deleting a branch (or a worktree whose branch - is gone) schedules its branch-DB removal through the daemon. A periodic - daemon sweep reconciles `branches/` against live git refs. `branch gc` - remains the manual verb; the automatic path is the default. - - **Dated amendment (2026-08-07, recorded decision, supersedes the - per-branch-DB model above).** Per-branch SQLite copies are retired as the - write-side mechanism. Non-default-branch writes now land in the single +1. **Branch store lifecycle.** Per-branch SQLite copies are retired as the + write-side mechanism (dated amendment, 2026-08-07). Non-default-branch + writes now land in the single project store's next branch-graph publication epoch, fenced by the sync lease and sealed through the branch-meta `graph_source` CAS (`feat(graph): fence branch graph publication and mutation epochs`, diff --git a/docs/plans/tracedecay-v2/39-embedded-grafeo-graph-database.md b/docs/plans/tracedecay-v2/39-embedded-grafeo-graph-database.md index c37a3d432b..ffb9ec389a 100644 --- a/docs/plans/tracedecay-v2/39-embedded-grafeo-graph-database.md +++ b/docs/plans/tracedecay-v2/39-embedded-grafeo-graph-database.md @@ -1,14 +1,5 @@ # Embedded Grafeo Graph Database Implementation Plan -> **SUPERSEDED BY LANDING (2026-08-13).** The checklist below predates the -> cutover and was never ticked; the work shipped regardless. Evidence: -> `crates/tracedecay-graph-db` is a workspace member pinned to Grafeo -> `=0.5.42` with `generation_runtime.rs` and the sealed-graph state digest; -> the daemon consumes it through -> `src/daemon/code_index_scheduler/graph_activation.rs`; and the SQLite graph -> authority was deleted in `79683eb527`. Treat the unchecked boxes as -> historical planning, not open work. - **Goal:** Replace custom adjacency structures and graph-shaped SQLite storage with one embedded Grafeo runtime boundary while retaining SQLite only for genuinely relational, transactional, and content-bearing records. **Architecture:** `tracedecay-graph-db` is the only workspace crate allowed @@ -20,9 +11,8 @@ content, and source manifests remain in their domain stores; Grafeo is the sole persisted and queried graph/vector projection over those sources. SQLite does not store shadow adjacency, graph indexes, or vector indexes. -**Tech Stack:** Rust 2024, the published Grafeo `=0.5.42` crates from -crates.io, TraceDecay domain/store ports, Tokio cancellation, Criterion, and -cargo-nextest. +**Tech Stack:** Rust 2024, the Grafeo crates, TraceDecay domain/store ports, +Tokio cancellation, Criterion, and cargo-nextest. ## Global Constraints @@ -35,9 +25,6 @@ cargo-nextest. `.grafeo.wal` sidecar while the file is open and checkpoints it into the database file on close. Do not inject an already-open file, replace Grafeo's storage engine, or add a private/secure filesystem adapter. -- Depend on the exact published `=0.5.42` release. Do not pin an unpublished Git - revision, maintain a TraceDecay fork, vendor Grafeo, or reimplement its - storage, WAL, checkpoint, or recovery machinery. - `tracedecay-graph-db` is the sole production direct `grafeo-*` dependency. - Do not introduce another persisted/query graph or vector database. A bounded in-memory algorithm is not a storage authority, but should use the standard @@ -79,8 +66,6 @@ Official implementation references: - [Grafeo repository and crate layout](https://github.com/GrafeoDB/grafeo) - [Embedded Rust API](https://grafeo.dev/user-guide/rust/) -- [Vector and hybrid search](https://grafeo.dev/user-guide/vector-search/) -- [Published Rust API for `0.5.42`](https://docs.rs/grafeo/0.5.42/grafeo/) - [Persistent storage and WAL](https://grafeo.dev/user-guide/persistence/persistent/) --- @@ -250,11 +235,10 @@ Expected: fail because the crate and interfaces do not exist. - [ ] **Step 2: Add the workspace crate and centralize Grafeo dependencies** -Use the exact published Grafeo `=0.5.42` crates in the new crate. Add -`tracedecay-graph-db` to `[workspace].members` and centralize those exact -versions in `[workspace.dependencies]`. Enable Grafeo's public `grafeo-file` -feature together with `wal`. No other workspace crate directly depends on -Grafeo, and no Grafeo dependency uses a Git source. +Use the workspace Grafeo crates in the new crate. Add `tracedecay-graph-db` +to `[workspace].members` and centralize those dependencies in +`[workspace.dependencies]`. Enable Grafeo's public `grafeo-file` feature +together with `wal`. No other workspace crate directly depends on Grafeo. - [ ] **Step 3: Implement typed open, snapshot, batch, traversal, and vector adapters** @@ -395,67 +379,6 @@ cargo clippy -p tracedecay-graph-db -p tracedecay-code-index -p tracedecay-query git commit -am "refactor(code-graph): route Grafeo through graph-db" ``` -## Task 4: Move semantic vectors and hybrid retrieval - -**Files:** -- Modify: `crates/tracedecay-semantic/src/projector.rs` -- Modify: `crates/tracedecay-semantic/src/runtime_query.rs` -- Modify: `crates/tracedecay-application/src/store/vector_generations.rs` -- Modify: `src/store/vector_generations.rs` -- Modify: `crates/tracedecay-query/src/retrieval/semantic/execution_authority.rs` -- Delete: SQLite code-semantic vector payload/state tables superseded by graph-db - -**Interfaces:** -- Consumes: `EmbeddingProjectionKey`, verified `EmbeddingVector`, source - generation, model/artifact digest, metric, dimensions, and normalization. -- Produces: recovered-state-verified vector publication and bounded Grafeo - vector similarity; TraceDecay composes hybrid retrieval above it. - -```rust -pub trait SemanticVectorStore { - fn publish_generation( - &self, - generation: &VectorGenerationId, - vectors: Vec, - cancellation: &CancellationToken, - ) -> Result; - - fn search( - &self, - request: &SemanticVectorSearchRequest, - ) -> Result; -} -``` - -- [ ] **Step 1: Add failing vector behavior tests** - -Cover cosine/dot/euclidean behavior as declared by the manifest, dimension mismatch, non-finite values, stale generation, cancelled query, filtered namespace, deterministic ties, reopen, and serving the previous generation during rebuild. - -- [ ] **Step 2: Move vector payloads and indexes to graph-db** - -Store the vector on the corresponding projected entity or a typed vector -entity linked to it. Keep model lifecycle/install metadata and publication -receipts relational. Publish vector and code graph generations in one graph-db -batch when they share a generation. - -- [ ] **Step 3: Implement hybrid retrieval above the storage primitive** - -Grafeo returns bounded vector candidates. `tracedecay-query` retains -BM25/lexical retrieval and performs authorization, fusion, normalization, -hydration, explanations, and stable cursors. - -- [ ] **Step 4: Delete SQLite vector payload duplication** - -Remove `semantic_vector_payload_v1`, `semantic_vector_state_slice_v1`, evaluation duplicates, and legacy vector blobs after every production reader moves. Retain only relational generation/model lifecycle records that are not duplicate vector authority. - -- [ ] **Step 5: Verify and commit** - -```bash -cargo nextest run -p tracedecay-semantic -p tracedecay-query -p tracedecay-application --no-fail-fast -cargo bench --bench code_search --no-run -git commit -am "refactor(semantic): move vector search to graph-db" -``` - ## Task 5: Move Git topology and evidence relations **Files:** @@ -958,30 +881,6 @@ recovered-digest proof in compact form with "relation scalar endpoints do not match native topology" while the same rows seal and serve in replay form. Toy-scale contracts pass, so the defect is scale- or ordering-dependent. -**Decision.** Live-container reclaim is `compact()` under the maintenance -lease, not a bespoke re-seal or a quarantine-and-rebuild: - -1. Reproduce the endpoint mismatch in the fork with a generation-scale - fixture (the sealed-store contract's `rich_manifest` grown to the failing - size), fix it there, and gate `COMPACT_ROUND_TRIPS_BYTES` on the recovered - digest proving equal in compact and replay form for that fixture. -2. Add a maintenance-cadence step beside `run_branch_compaction`: when a - project's live container exceeds twice the summed bytes of its verified - heads' sealed artifacts (both figures are already censused, see the - `RetentionBacklog` finding's sealed evidence), take the project store - maintenance lease, quiesce the graph owner, `compact()`, close, and reopen. - The reopen reconstructs the layered store from the `CompactStore` section; - later writes land in the overlay, so the store stays mutable. -3. Prove the swap with the existing recovered-digest verification before the - compacted container replaces the live file; a mismatch keeps the original - and reports a typed degradation. Vector generations, whose only durable - home is the live container, are covered by the same proof and never - re-embedded. - -**Non-goals.** No re-projection from canonical replay authorities as a -compaction path (that is corruption recovery and would re-embed vectors), and -no quarantine of a healthy container. - ### Correction (2026-09-12, later the same day, measured) The decision above rests on a wrong premise. Read against the pinned fork: @@ -1039,14 +938,10 @@ The decision above rests on a wrong premise. Read against the pinned fork: base rather than a replay into RAM, deletes land in the overlay's deletions section, and `recompact()` merges on the maintenance cadence. Blocked on validating TraceDecay's staging writes against a - `LayeredStore` (README gaps 3 and 4: `has_vector_index` planning and - the MVCC-free base). + `LayeredStore` (README gap 4, the MVCC-free base). b. Rebuild a fresh container by streaming the retained authorities, sealed heads and their dependency closure from their compact stores, - pending replays from `canonical_replay_source`, vector generations - from their sealed stores, then prove every generation's recovered - digest against the journal before swapping files under the - maintenance lease. Memory is bounded by the largest generation, not - the container. Blocked on vector generations: the sealed lane never - routes vector search to sealed stores, so a rebuilt live container - must re-stage the vector rows and rebuild the HNSW index on open. + and pending replays from `canonical_replay_source`, then prove every + generation's recovered digest against the journal before swapping + files under the maintenance lease. Memory is bounded by the largest + generation, not the container. diff --git a/docs/plans/tracedecay-v2/40-partitioned-generation-storage-and-zero-copy-replay.md b/docs/plans/tracedecay-v2/40-partitioned-generation-storage-and-zero-copy-replay.md index 2cdba660c4..36e520ba0c 100644 --- a/docs/plans/tracedecay-v2/40-partitioned-generation-storage-and-zero-copy-replay.md +++ b/docs/plans/tracedecay-v2/40-partitioned-generation-storage-and-zero-copy-replay.md @@ -1,16 +1,5 @@ # 40. Partitioned generation storage and zero-copy replay retirement -Status: ACTIVE (decided 2026-08-22). Supersedes the monolithic sealed-generation -envelope as the long-term storage design; stages 0–1 land first as compatible -fixes inside the current `format_revision`. - -Stage state (reconciled 2026-08-31 against the tree, not the commit log): -stages 0a, 0b, and 0c are landed; stages 1 and 2 are landed; stage 3 is -untouched. Partitioned writers emit `format_revision` 7 while readers retain -explicit revision-5/6 monolithic compatibility. The -§Problem measurements below describe the pre-stage-0 state and are retained -as the original evidence, not as a description of the current tree. - ## Problem (measured, 2026-08-21/22, this repository) - One sealed code generation for this repository is a single @@ -63,55 +52,6 @@ representation and the replay economics: 5. **No second whole-generation copy, ever.** Replay reads canonical artifacts; retention must never require a byte-duplicating insurance copy. -## Staged delivery (each stage ships alone, oldest-first compatible) - -- **Stage 0, zero-copy replay retirement** (this change set, three commits): - - **0a** `hydrate_sealed_code_generation` resolves the sealed payload from - the canonical `code-generations-v1/` root first and falls back to the - replay pool. Readers become location-agnostic; digest verification on read - makes both locations equally trustworthy. Retirement moves strictly - canonical→pool, so a canonical-then-pool probe cannot miss a live file. - - **0b** Retention retires a superseded generation by **atomic rename into - the replay pool** (no-clobber; content-identical `AlreadyExists` collapses - to unlink) instead of unlinking, whenever the graph projection has not yet - durably consumed it. Rename is metadata-only and crash-atomic: the file is - always in exactly one of the two roots. The existing release queue keeps - deleting pool entries once the graph append is durable. - - **0c** Delete the eager staging copy from the graph publish path - (`install_project_graph_replay_seal_at` / `stage_project_graph_replay_seal` - and the copy machinery in `seals.rs`). Steady state stages **zero bytes**; - the pool only ever holds generations that were retired while still needed. -- **Stage 1, single-parse activation**: graph publication consumes the - already-decoded active generation (the projection manifest already rides - along on first publication; extend the same guarantee to the recovery - branches via the Stage-0a fallback) so cold activation parses the sealed - payload at most once. -- **Stage 2, manifest + segment split** (`format_revision` bump): split the - sealed envelope per §Decision 1–2; integer-keyed, content-addressed - segments; the 2 GiB whole-envelope bound and whole-file rewrite per - generation disappear; unchanged-file segments are shared across - generations. -- **Stage 3. Grafeo startup authority**: manifest ↔ Grafeo head validation - replaces replay on clean startup per §Decision 3–4. - -## Invariant mapping (old mechanism → new mechanism) - -| Invariant | Today | After | -|---|---|---| -| Content addressing | digest-named monolithic JSON | digest-named manifest + digest-named segments | -| Atomic publish | temp+rename of envelope + pointer | segments/Grafeo commit first, then manifest temp+rename | -| Crash recovery | eager 1.39 GB pool copy + journal replay | rename-retired canonical files + scoped lazy replay | -| Retention | unlink source; release pool copy via queue | rename-to-pool when graph pending; queue unchanged | -| Verification | full-file digest on every decode | per-component digest on read (unchanged per file) | -| Branch identity | durable index entry evidence | unchanged (manifest carries it) | - -## Non-goals - -- No change to sealing semantics, projector revisions, or the relational - journal contract in Stage 0. -- Stage 0 does not change `format_revision`; on-disk envelopes stay readable - both directions across the stage-0 commits. - ## Dated amendment (2026-09-12, recorded decision): stage 2b, shared, compressed segments **Measured.** For one worktree of a 150 MB / 5,190-file repository the scoped From 81e9db457932e4da63b6aa1f727d56d87b13a5ec Mon Sep 17 00:00:00 2001 From: Zack Jackson <25274700+ScriptedAlchemy@users.noreply.github.com> Date: Thu, 5 Nov 2026 22:22:39 -0800 Subject: [PATCH 2/2] docs(storage): state that store budgets are not enforced yet Removing the Plan 38 reference left a target reading as a current fact. `Storage::maintain` does not exist in the code, and an over-budget store only produces a Doctor finding. Co-Authored-By: Claude Opus 5.5 --- docs/plans/storage/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/plans/storage/README.md b/docs/plans/storage/README.md index e1ad159665..323ccda22a 100644 --- a/docs/plans/storage/README.md +++ b/docs/plans/storage/README.md @@ -622,7 +622,8 @@ changes a file that holds memory. - Plan 02's "No asynchronous database API" and its interactive repository transactions: every storage API is async and every transaction is one job. -- Store soft budgets have defaults and are enforced by `Storage::maintain`. +- Store soft budgets: enforcement through `Storage::maintain` is a target that + does not exist yet. Today an over-budget store only produces a Doctor finding. - Plan 39's move of graph-shaped data off SQLite: mutable graph data (memory and session relations) lives in SQLite; Grafeo remains an immutable sealed format. - Plan 40's monolithic per-worktree generation segments: segments are