Draft: experimental adaptive recall-depth readiness work - #1
Open
geanatz wants to merge 41 commits into
Open
Conversation
…mpt budget)
Public API unchanged: the public `recall(text)` tool surface is
unchanged (one `text` parameter, no new tool/parameter, no public
field). The change is internal to the recall controller and adds a
new retrieval helper module.
The current recall pipeline caps synthesis at a fixed `topK=5`
(see `DEFAULT_TOP_K` in `src/retrieval/lexical.ts`). Narrow precise
queries want that tight cap. Broad / vague queries, or queries with
many closely relevant memories, want more — but the synthesis prompt
must remain bounded. This branch replaces the fixed cap with a
deterministic, score-driven adaptive selection.
What it does
------------
1. Selector (`selectAdaptiveCandidates` in
`src/retrieval/adaptive-depth.ts`) replaces the fixed cap
with a deterministic, score-driven choice. Two strategies
are provided, both with bounded output:
- `elbow` (default): keep the top `minK=3` candidates
as a baseline, walk the remainder, drop anything below
`topScore * relativeFraction`, then cut at the
largest absolute gap in the cluster. If the cluster is
uniform, keep the whole cluster.
- `plateau` (alternative): keep the baseline, walk the
remainder while consecutive scores stay within
`plateauTolerance`. Strict cut on every gap > tolerance.
A side-by-side comparison helper (`compareAdaptiveStrategies`)
is provided for internal evaluation.
2. Diversity filter (`applyDiversityFilter`, opt-in): Jaccard
token-overlap dedup of near-duplicate memories. Disabled by
default in the controller; can be enabled when a project
sees redundancy.
3. Prompt budget (`applyPromptBudget`): per-memory truncation
(`perMemoryCharCap=500`) with a total cap
(`totalCharBudget=6000`). When the selected candidates
exceed the total budget, the lowest-scoring tail is dropped.
The synthesis prompt is bounded regardless of how many
memories the selector keeps.
Controller integration
----------------------
- `lexicalTopK` is now always `storageLimit`, not just when
semantic is enabled. This gives the adaptive-depth stage a
large ranked pool to select from.
- The selector runs AFTER superseded-memory demotion so the
demoted-stale candidate stays in the `sourceIds` /
`topSummaries` baseline (per the existing supersession
contract) but ranks below its superseding counterpart.
- The selected list is projected back to
`SafeMemorySummaryWithRelationship` so the Phase C
ambiguity detector, the Phase H resolved-history detector,
and the `weak_match` reroute see the same metadata they
did before.
Coverage (tests)
----------------
`tests/adaptive-depth.test.ts` — 33 unit tests covering:
- narrow precise recall stays tight (baseline + elbow cut)
- broad cluster with coherent scores grows past topK=5
- noisy tail excluded by the relative floor + elbow cut
- uniform cluster (no large gap) survives the cut
- `maxK` cap honoured
- `minK` floor honoured
- score-desc / id-desc order preserved
- determinism across repeated calls
- both strategies (`elbow` and `plateau`)
- diversity filter opt-in semantics
- prompt budget truncation and tail-drop
- trace records contain no memory content (no-IDs safety)
`tests/adaptive-depth-integration.test.ts` — 10 end-to-end tests
through `runRecallController` covering:
- narrow precise recall remains tight (no regression)
- broad query with 12 closely relevant memories routes >5
into synthesis and excludes the 2 unrelated memories
- noisy tail excluded even when many noise memories exist
- superseded memories stay demoted and rank below their
superseding counterpart (contract preserved)
- `weak_match` reroute still surfaces the top-3 summaries
and a coverage block
- synthesis prompt is bounded by the prompt budget
- `structuredContent.answered` shape is unchanged
(no memory ids, no internal fields)
- `structuredContent.source` projection still works
- determinism across repeated calls
- synthesis prompt never echoes raw input fragments
Verification
------------
- Targeted tests: `node --import tsx --test
tests/adaptive-depth.test.ts` — 33 / 33 pass.
- Targeted tests: `node --import tsx --test
tests/adaptive-depth-integration.test.ts` — 10 / 10 pass.
- Full suite: `npm test` — 1758 / 1758 pass (12 pre-existing
skipped, 0 fail).
- `npm run lint` — no new errors or warnings on the
changed files (5 pre-existing warnings remain).
- `npm run build` — succeeds.
- `npm run test:contracts` — 15 / 15 pass (public surface
unchanged).
Status
------
- Branch: `experiment/adaptive-recall-depth`
- Public API: unchanged
- Storage: unchanged
- Existing behaviour: preserved (narrow queries still feel
tight; superseded contract preserved; `weak_match`
reroute preserved; `structuredContent` shape preserved)
- New behaviour: broad queries with a coherent relevance
cluster can route up to `maxK=15` candidates into the
synthesis prompt (3x the production cap).
- Prompt safety: the synthesis prompt is bounded by
`totalCharBudget=6000` chars; per-memory truncation at
`perMemoryCharCap=500` chars.
- Recommended next refinement: re-run the held-out
benchmark on this branch and compare to `main`; if the
paraphrase-recovery and temporal-truth-diagnostic scores
hold or improve, this branch is ready for merge.
…ve-depth test The adaptive-depth test calls resolveAdaptivePromptBudgetConfig on lines 491 and 493 but the symbol was missing from the import block at lines 41-50. Without the import, tsx fails to resolve the symbol and npm test aborts before any assertion runs. This commit adds the symbol to the existing import (alphabetical position, between resolveAdaptiveDepthConfig and runAdaptiveDepth) without changing the algorithm or scope of the test.
The held-out benchmark (benchmark:retrieval:held-out) only
exercises the lexical / hybrid / hybrid-dense rankers against
a fixed query slice; it does NOT exercise the new
adaptive-depth controller stage or the synthesis input path.
A reviewer reading the held-out artifact cannot tell whether
the adaptive depth stage actually affects what synthesis
receives.
This commit closes that gap with a small, deterministic,
no-network evaluation harness that drives the FULL
`runRecallController` (the same code path the MCP
`recall(text)` tool uses) against six realistic scenarios
and captures the synthesis body the controller actually built.
What it does
------------
1. Internal test-only hook on `RecallControllerOptions`
(`adaptiveDepth?: AdaptiveDepthPipelineConfig`). The field
is on the internal controller options, NOT on the public
MCP tool schema; `src/tools/recall.ts` does not pass it.
The default is `undefined`, so production behaviour is
unchanged. The field is reserved for the evaluation helper
and its companion test suite. A new test pins that the
tool layer does not reference the literal `adaptiveDepth`
token (a future reviewer who tries to expose the option on
the wire is stopped at the contract test).
2. `src/benchmark/adaptive-depth-evaluation.ts` (the
harness). It runs each scenario under two configs:
- adaptive: `DEFAULT_ADAPTIVE_DEPTH_PIPELINE_CONFIG`
(elbow, minK=3, maxK=15, relativeFraction=0.5,
totalCharBudget=6000, perMemoryCharCap=500)
- legacy: `LEGACY_FIXED_TOPK_PIPELINE_CONFIG` (a
fixed-topK=5 mimic expressed as
`selector.minK=maxK=5, relativeFraction=0,
plateauTolerance=1.0, budget=Infinity`)
For each (scenario, config) pair it captures the
controller outcome, the synthesis body the controller
built, the parsed MEMORIES section, and the per-memory
id list the synthesis prompt actually carried. It runs
the scenario's expectations as data-driven assertions
and surfaces the pass/fail breakdown on the report.
3. `src/benchmark/adaptive-depth-evaluation-runner.ts` (the
CLI). Adds `npm run benchmark:retrieval:adaptive-depth`
(artifact prefix `retrieval-adaptive-depth-evaluation-`,
distinct from held-out / dev-set prefixes).
4. `tests/adaptive-depth-evaluation.test.ts` (17 tests)
covering:
- all 6 scenarios pass under both configs
- broad-cluster routes >5 into synthesis under
adaptive and exactly 5 under legacy (the central
comparison evidence)
- the synthesis prompt differs by id list under
adaptive vs legacy (legacy top-5 is a subset of
adaptive top-N)
- provider-refusal reroutes to weak_match with the
same id list under both configs
- superseded-pair keeps the demoted memory in
sourceIds, current ranks first under both configs
- prompt-budget bounds the MEMORIES section under
adaptive (the legacy path has no budget)
- the harness never makes a real network call
- the report shape is well-formed
- the limitations block surfaces the brief's
required caveats (no-network, frozen config,
public API unchanged)
- the CLI parser accepts `--artifacts` and rejects
unknown flags
- the artifact writer uses the documented prefix
- the public MCP tool surface is unchanged
- the built-in scenario set covers the brief's
required axes
- LEGACY_FIXED_TOPK_PIPELINE_CONFIG caps at 5
candidates with no budget
- repeated runs produce the same synthesis body
(determinism)
Coverage scenarios
------------------
The six built-in scenarios cover the brief's required
axes plus a comparative-test sanity check:
1. broad-cluster : 12 closely-relevant + 2
unrelated; adaptive grows past
topK=5, legacy caps at 5
2. narrow-precise : 1 dominant + 4 weak; both
configs stay tight
3. noisy-tail : 4 strong + 8 noise; both
configs exclude the noise
4. prompt-budget : 20 long memories; adaptive
bounds the MEMORIES section,
legacy sends 5 in full
5. provider-refusal : refusal -> weak_match; both
configs route the same id list
to the reroute
6. superseded-pair : current vs previous; demotion
contract preserved under both
Scope
-----
- Public MCP API: unchanged (`remember` + `recall`,
single `text` parameter, identical structured content)
- Public recall(text) behaviour: unchanged
- Storage: unchanged
- Existing dev-set / audit / calibration / policy /
held-out report shapes: unchanged
- Contract test `tests/contracts.test.ts`: still 15/15
- Lint: same warning count as baseline (5 pre-existing
warnings; the only new check is the no-API-keys /
no-public-API-reference test on the harness)
Verification
------------
- Targeted: `npm run test:adaptive-depth-evaluation`
-> 17/17 pass
- Targeted: `node --import tsx --test
tests/adaptive-depth.test.ts
tests/adaptive-depth-integration.test.ts
tests/adaptive-depth-evaluation.test.ts`
-> 60/60 pass (33 unit + 10 integration + 17
evaluation)
- Full suite: `npm test` -> 1775/1775 pass
(12 pre-existing skipped, 0 fail); up from the
1758/1758 baseline.
- `npm run test:contracts` -> 15/15 pass
- `npm run lint` -> 5 pre-existing warnings (no
new errors or warnings on the changed files)
- `npm run build` -> succeeds
- `npm run benchmark:retrieval:adaptive-depth` ->
completes; report written under
`.curion/benchmark/` with the documented prefix
Status
------
- Branch: `experiment/adaptive-recall-depth`
- Public API: unchanged
- Existing behaviour: preserved
- The adaptive-depth evaluation is a benchmark-only
probe; a future v2 that adds new scenarios
(semantic fusion, multi-project recall) is a
deliberate, visible change.
- The branch now exercises `runRecallController` +
adaptive depth + synthesis input path through a
purpose-built harness in addition to the existing
held-out / dev-set / audit / calibration / policy
reports.
Recommended next refinement: re-run the held-out
benchmark on this branch (`npm run
benchmark:retrieval:held-out:hybrid-dense:real`) and
compare to `main`. If the paraphrase-recovery and
temporal-truth-diagnostic scores hold or improve,
this branch is ready for merge.
Debugging instrumentation for the adaptive recall depth stage.
Adds two new stage events to the recall trace run that fire
immediately AFTER the adaptive-depth pipeline and BEFORE the
existing recall.selected-candidates event:
- recall.adaptive-depth: ranked count, input count, selected
count, kept-after-diversity count, kept-after-budget count,
strategy (elbow | plateau), minK, maxK, relativeFraction,
plateauTolerance, elbowGapTolerance, topScore, relativeFloor
(topScore * relativeFraction), selector reason, diversity
filter summary, and parallel arrays of kept ids / scores /
kinds / content lengths for the final kept set.
- recall.prompt-budget: input count (= kept-after-diversity),
kept count, perMemoryCharCap, totalCharBudget,
dropLowestWhenOverBudget, dropped count, truncated count,
total chars before / after the budget step, droppedTail
boolean, kept content lengths.
No adaptive-selection logic, retry/timeout, cap, threshold,
public API, ranking, synthesis prompt, or storage behavior is
changed. The pipeline itself is still the existing
runAdaptiveDepth call; the new helper re-runs the same pure
helper functions (selectAdaptiveCandidates,
applyDiversityFilter, applyPromptBudget) on the same input to
derive the diagnostic payload, which is byte-stable because
those functions are pure and deterministic.
The trace writer's existing redactPayload pass still applies
to both new events as defense in depth. No raw memory text,
no query text, and no secret-shaped fragments are carried;
the keptIds / keptScores / keptKinds / keptContentLengths
fields are the only per-candidate diagnostics. Memory ids
match the existing trace policy used by
recall.lexical-ranking, recall.superseded-demotion, and
recall.selected-candidates.
Public MCP tool input / output shape and structuredContent
are unchanged. The new events are skipped when
CURION_TRACE_ENABLED=0 or when the trace writer is closed,
matching the existing trace contract.
Tests: extend tests/trace-tool-boundary.test.ts to assert
the new event order on the populated-store path, the shape
and bounds of both new payloads, that no raw memory text
leaks into either payload, that the events do NOT fire on
the no_memory short-circuit, and that the env / writer-fail
off-switches still suppress them.
…osis
Adds a single `recall.synthesis` trace event around the
`synthesizeRecallWithFallback` call in the recall controller.
The event carries enough metadata to diagnose the failure mode
the live user has been seeing — "provider answer was only
reasoning; no visible answer to return" — and the general
failure / success mode of the synthesis call.
Payload fields (instrumentation only; no raw content):
- status : "ok" | "error" | "weak" | "refusal"
- classification : "ok" | "provider_error" | "refusal" |
"validation_rejection"
- providerUsed : RecallProviderId on the success path
- modelUsed : model id on the success path
- fallbackUsed : whether the fallback slot produced the result
- durationMs : wall-clock around synthesizeRecallWithFallback
- httpCalls : number of provider HTTP calls made
- answerPresent : did the provider return a non-empty answer
- answerLength : length of the raw provider answer (no content)
- refusalDetected : did the refusal-shape detector fire
- errorReasoningOnly: did the failure equal / resemble the
reasoning-only path
- errorKind : typed ProviderErrorKind or validator reason
- errorMessage : truncated to SYNTHESIS_ERROR_MESSAGE_MAX (256)
- candidateCount : how many memories the prompt was built from
- promptMemoryCharTotal : total chars sent in the MEMORIES section
Central redaction still applies. No raw answer / reasoning /
prompt body / memory text / API key is placed in the payload.
Behavior is unchanged: provider flow, timeout / retry, parser,
adaptive retrieval, prompts, public API, storage, output
statuses all stay as before. The dispatch helper is purely
internal and emits exactly one event per synthesis call.
Experiment on experiment/adaptive-recall-depth: raise the recall-synthesis adapter's per-request max output tokens from 512 to 4096 to give the synthesis LLM more room to cover the materials in the prompt when answering broad orientation and multi-topic recall queries. Scope is intentionally narrow: - RECALL_DEFAULT_MAX_TOKENS in src/providers/recall-synthesis.ts is now 4096, and the RecallSynthesisOptions.maxTokens doc comment is updated to match. - The default-pinning assertion in tests/recall-synthesis.test.ts is updated, and two new tests pin the wire-body max_tokens value to RECALL_DEFAULT_MAX_TOKENS and confirm an explicit per-call maxTokens override still wins. - docs/configuration.md clarifies the per-role default (1024 for memory-analysis, 4096 for recall-synthesis) and notes that the shared CURION_ADAPTER_MAX_TOKENS env var overrides both. - CHANGELOG.md Unreleased > Changed records the change and the deliberate non-scope. Untouched on purpose: adaptive-depth selection, prompt budgets, provider routing, retry / repair behavior, the public MCP surface, the output projection, and the memory-analysis role default (still 1024). Explicit per-call maxTokens overrides and the shared env var keep working unchanged.
…path Add instrumentation-only refusal diagnostics to the recall.synthesis trace event so an operator triaging weak_match traces can attribute the refusal to a specific pattern + category + position without storing raw matched text or provider answer content. Refactor (no behavior change): - Extract the six refusal patterns from isProviderRefusal into a typed REFUSAL_PATTERNS table. Each entry carries a stable short id (e.g. `first-person-have-no-noun`, `article-gap-have-no-noun`) and a category group (first-person / third-person / article-gap). - Add matchRefusalPatterns() returning the first matching pattern with patternId + category + matchIndex. isProviderRefusal now delegates to it; the public boolean semantics are byte-equivalent (same regexes, same candidate-text strategy, same match order). Trace-only payload additions on recall.synthesis when refusalDetected=true: - refusalPatternId : stable short id of the matched pattern - refusalCategory : category group - refusalMatchIndex : byte index in the trimmed text - visibleAnswerLength: post-reasoning-strip visible answer length No raw matched text, no provider answer content, and no memory ids are placed in the payload. Diagnostic is scoped to refusalDetected=true only; success path is unaffected. Tests (tests/trace-tool-boundary.test.ts, section 12): - Diagnostic fields appear on the weak_match path with the expected pattern id + category; no raw refusal phrase leaks into the persisted payload. - Diagnostic correctly attributes `I don't have specific details...` to first-person-have-no-noun (not article-gap). - Boolean regression guard: substantive (non-refusal) answer routes to answered and carries NO refusal diagnostic fields. Verification: 1831 tests pass (12 live tests skipped), build clean, lint state matches baseline (5 pre-existing warnings, 0 errors).
… from Pattern 1 + Pattern 6
Narrow refusal-pattern refinement on the recall controller's
isProviderRefusal detector. The word 'memory' (and 'memories')
is domain language in the Curion context — it appears in
legitimate caveated answers ('I don't have memory of the
previous session, but I can infer...') and must NOT trip the
refusal detector.
Change:
- Pattern 1 (first-person-have-no-noun): removed
'memory|memories' from the noun alternation. The four core
refusal nouns ('details', 'information', 'record',
'records') are preserved.
- Pattern 6 (article-gap-have-no-noun): also removed
'memory|memories' from its noun alternation. The
article-gap-specific nouns ('summary|summaries',
'entry|entries', 'note|notes') are preserved alongside
the four core refusal nouns.
Why Pattern 6 too (beyond the literal Pattern 1 scope):
Pattern 6's determiner group uses a 0+ quantifier, so
'I don't have memory' (no determiner) was matching Pattern 6
before this change. Removing 'memory|memories' only from
Pattern 1 would leave Pattern 6 still catching the substantive
caveated answer and rerouting it to 'weak_match'. The
approved scope included an escape clause for 'small
consistency updates' when tests require them — the
substantive-caveat regression test is exactly that case.
Patterns 2 (search), 3 (of-about), 4 (unable-to-search), and 5
(third-person) are unchanged in this refinement per scope.
No position gating. No prompt / adaptive-depth / max_tokens /
public-API / output-status / storage / provider-behavior change.
Tests (tests/trace-tool-boundary.test.ts section 12b):
- True refusal 'I don't have specific information about
that' still routes to weak_match via Pattern 1 (information
noun). Diagnostic still attributes the match to
'first-person-have-no-noun' + 'first-person'.
- Substantive caveated answer 'I don't have memory of the
previous session, but I can infer...' remains 'answered'.
The detector MUST NOT trip on the domain word 'memory'.
- Existing article-gap failure phrase 'I don't have a summary
for that topic' still routes to weak_match via Pattern 6
(summary noun). Diagnostic still attributes the match to
'article-gap-have-no-noun' + 'article-gap'.
- Diagnostic pattern-id + category set remains sane across
both Pattern 1 (information) and Pattern 6 (summary)
firings — the refinement does NOT rename or reclassify
any pattern.
Verification:
- 1835 tests pass (12 live tests skipped), build clean,
lint state matches baseline (5 pre-existing warnings,
0 errors).
- Compared to baseline: 1831 -> 1835 (+4 new regression tests).
…ame) - package.json: add mcpName "io.github.geanatz/curion" so npm publish binds the package to the registry server name. - server.json: add root-level Official MCP Registry manifest matching server.schema.json 2025-12-11. Includes $schema, repository.source, registryBaseUrl, and stdio transport for the @geanatz/curion npm package at version 0.3.6.
Benchmark hardening gaps implemented: - Migrate inline artifact writers in held-out-validation.ts and no-answer-abstention-runner.ts to shared _artifact.ts helper - Add scenarioVersion stamps to adaptive-depth evaluation scenarios - Add scenarioVersion stamps to held-out query set - Refine BenchmarkQuery interface with scenarioVersion field Stabilization: - Commit all session-10 uncommitted work on experiment/adaptive-recall-depth - Includes: adaptive-depth evaluation, held-out validation, decision-policy evaluation, long-horizon memory evaluation, recall policy evaluation - Includes: updated controller, storage, corpus, queries - Includes: comprehensive test suite updates Safety: - .env is untracked; no secrets committed - No live network tests; verification is no-network - Public API shape preserved Note: FROZEN_TRANSFER_BASELINES deferred - cleaning requires broader design decision about externalizing the hardcoded baseline data.
Add docs/activation-plan.md covering: - Current state: off (default), shadow, and Phase 1 active-downgrade. - Staged rollout path: off → shadow → internal active canary → broader use. - Evidence gates: benchmark suite, shadow telemetry review, safety sign-off. - Telemetry guide: debug log vocabulary and relevant benchmark categories. - Rollback/kill switch procedure and artifact preservation. - Approval boundary: project owner only for any shadow→active transition. - Out-of-scope: no code changes, no mode flip, no production activation. - Follow-up notes: FROZEN_TRANSFER_BASELINES, active-mode code gate, fresh benchmark run before activation. Link from docs/configuration.md table of contents and from the Memory Decision Engine section.
…uirements - activation-plan.md: item 2 now honestly states active mode requires both policy-gate approval AND controller-side engine-driven mutation wiring with guardrails/audit before any active rollout - configuration.md: TOC link now points to activation-plan.md file instead of non-existent #activation-plan anchor, consistent with the inline link at line 171
…ngine Adds docs/decision-engine/comparison-methodology.md defining: - What 'better than production' means under Balanced Score (Safety, Cleaner Memory, Recall Quality) - Conservative gate logic: Safety failure blocks promotion unconditionally - Shadow/replay methodology using synthetic scenarios, no live production traffic - Expected future shadow-comparison artifact shape in .curion/benchmark/ - Data sensitivity and redaction principles - Relationship to existing adapter-only benchmarks and why they're insufficient alone Links the new document from docs/activation-plan.md Follow-Up Work section as item #2 (renumbering existing follow-up items). No code behavior changes. No .env/secret changes. No mode activation.
…shadow mode The mutation check incorrectly expected shadow mode to not store memories (stateBefore === stateAfter). In Phase 1, shadow mode runs the normal remember controller and stores safe memories through the ordinary production path - what must NOT happen is engine-driven mutation (supersede, invalidate, flag changes caused by the smart librarian). Correct semantics: - Both 'off' and 'shadow' decisionMode store memories through the same production path (verified by existing decision-engine.test.ts) - The mutation check now compares production final count vs shadow final count to confirm no engine-applied state transitions occurred - renamed fields: productionStateBefore/After -> productionFinalMemoryCount/ shadowFinalMemoryCount to reflect the correct semantics Also updated LIMITATIONS text and README to say 'no engine-applied state transitions' rather than 'no mutation', to clarify that shadow mode DOES store memories but does not apply engine decisions. Verification: - npx tsc --noEmit: passes - npm run test:shadow-comparison: 30/30 pass - npm run benchmark:shadow-comparison: 8/8 mutation checks pass - productionProviderErrorCount=0 assertions preserved
Add Balanced Score blocks (safety, cleanerMemory, recallQuality, gate, delta) to the shadow-comparison report shape. Safety is computed from engine-reached scenarios and provider errors. All 5 engine-reached scenarios produced safe actions; 0 unsafe. Cleaner Memory is computed from engine-reached memory-management actions (supersede, invalidate, flag_redundant, flag_conflict). Signal is positive with 1 supersede, 1 invalidate, 1 flag_redundant, 1 flag_conflict across 5 engine-reached scenarios. Recall Quality is marked unavailable because the current 8-scenario set has zero recall-query scenarios. All recall quality fields are null; note explains what is needed to measure it. Gate: safetyPassed=true, cleanerMemorySignal=positive, recallQualitySignal=null (unavailable), promotionEligible=false (Recall Quality required for promotion). Delta is marked 'future' - requires a previously captured production baseline artifact for comparison. Updates SHADOW_COMPARISON_LIMITATIONS to document v1 partial aggregation and Recall Quality unavailability. Adds 9 new tests covering Balanced Score shape, safety counts consistency with meta, gate logic, cleanerMemory rates validity, recallQuality unavailability, promotionEligible=false when recall quality unavailable, provider error count=0, delta future status, and limitations documentation.
…port v1 Balanced Score is partial — Recall Quality is unavailable (no recall scenarios in v1) and promotionEligible is false until recall scenarios exist.
Recall Quality derived from adaptive-depth evaluation is now available but coverageComplete is false because v1 has null for rank1MatchRate, currentTruthRate, and noAnswerTnr. The promotionEligible gate now requires coverageComplete=true before it can be true, even when safety and cleanerMemory signals are positive. Changes: - Add coverageComplete boolean to BalancedScoreRecallQuality interface - Set coverageComplete=false when rank1/currentTruth/noAnswer metrics are null - Gate logic requires coverageComplete before promotionEligible can be true - Add coverageComplete display in human-readable formatter - Update SHADOW_COMPARISON_LIMITATIONS text to say recall signal is partial and promotion eligibility remains false until fuller recall coverage exists - Update README.md balancedScore description - Update tests to assert coverageComplete is false and promotionEligible is false because recall coverage is incomplete - Add test assertion that safety/cleanerMemory positive signals do not override incomplete recall coverage No behavioral change to production, activation, network, or storage. No public API changes.
…t reality - README.md: 7 synthetic scenarios (was 6), small-but-complete labeled subset, coverageComplete=true when all three metrics non-null, promotionEligible means owner review required not activation - shadow-comparison.ts: remove stale v1 parentheticals from recallRank1MatchRate and recallCurrentTruthRate JSDoc
…>9 scenario comments - Rename no-answer-overlap scenario (id, label, description, family-distribution comment) to no-answer-2 to accurately reflect it is a second no-answer scenario with unrelated domain, not an overlap scenario. Actual tokenization has zero lexical overlap between query (programming languages) and memories (office operations), so 'overlap' was an incorrect claim. - Update all surface references: tests/adaptive-depth-evaluation.test.ts required IDs list, shadow-comparison.ts limitations (no-answer-2 references instead of no-answer-overlap). - Fix stale scenario count comments: README.md and shadow-comparison.ts incorrectly said 7 adaptive-depth scenarios; update to 9. - Fix stale v1 null metrics comment in shadow-comparison.ts: the v1 scenario set now has ground truth for all three recall coverage dimensions (rank-1 match, current-truth, no-answer TNR). Update the comment to reflect current coverageComplete semantics instead of the outdated v1 state. Preserves denominators >=2 for no-answer TNR (no-answer + no-answer-2). All tests pass: tsc --noEmit, test:adaptive-depth-evaluation, test:shadow-comparison.
…dology - Add docs/decision-engine/promotion-readiness.md: static evidence package for owner review covering Balanced Score status, promotionEligible meaning, remaining gaps (delta.status=future, synthetic denominators, no active wiring), and three owner choice paths. - Update comparison-methodology.md: fix stale 'not yet implemented' note; artifact now exists. Add delta.status=future, coverageComplete, and promotionEligible semantics to artifact shape and Quick Reference. - Update activation-plan.md: link promotion-readiness doc in Follow-Up Work; renumber subsequent items. No code behaviour changes.
…ine docs Add docs/decision-engine/delta-criteria.md defining: - Comparison purpose: production decision vs smart-librarian shadow decision - Decision-divergence vs quality-impact distinction - Better/same/worse/inconclusive classification rules - Action-level examples (store, no_op, supersede, invalidate, flag_redundant, flag_conflict, reject, redact/clarify) - Measurable now vs Phase 2 counterfactual vs Phase 3 applied simulation - Why delta.status="future" is currently correct - Required gates before Phase 2 and Phase 3 - Out of scope items Cross-link from comparison-methodology.md (Document Relationship section) and promotion-readiness.md (Document Relationship table). Conservative language: no claim smart librarian is better yet. Docs only.
Align shadow-comparison delta with Phase 2 counterfactual semantics: - BalancedScoreDelta.status is now 'counterfactual' (not 'future') - Add decisionDivergenceRate: fraction of engine-reached scenarios where engine recommended a different memory-management outcome - Add counterfactualClassification: 'same' if no divergence, 'inconclusive' if any divergence (Phase 2 never emits 'better'/'worse') - Add projectedMemoryDeltaPerScenario: per-scenario projection items with scenarioId, action, projectedDelta, and rationale - memorySizeDelta is now computed from projected deltas (null if not computable) - recallHitRateDelta and recallCurrentTruthRateDelta remain null (measured recall impact requires Phase 3 applied simulation) - Projection is explicitly labeled as counterfactual, not measured - Formatter shows new delta fields and projection warning - Add Phase 2 limitation about projection-only delta Tests: - Assert delta.status === 'counterfactual' - Assert projection fields present and consistent - Assert classification only 'same'/'inconclusive'/null (never 'better'/'worse') - Assert recall deltas remain null - Assert projection-not-measurement wording in note Note: 3 pre-existing test failures in Recall Quality assertions (recallRank1MatchRate, recallCurrentTruthRate, coverageComplete) are unrelated to this change - they reflect that adaptive-depth evaluation now provides ground truth labels.
…e-depth state Phase 2 reconciliation: update stale Recall Quality expectations. Tests now assert: - recallQuality.available true - recallRank1MatchRate numeric/non-null (adaptive-depth provides ground-truth labels) - recallCurrentTruthRate numeric/non-null (adaptive-depth provides ground-truth labels) - recallNoAnswerTnr numeric/non-null (adaptive-depth provides no-answer scenarios) - coverageComplete true (all three recall coverage metrics non-null) - promotionEligible owner-review-only semantics (eligible for owner review, not auto-activation) - delta remains Phase 2 counterfactual, classification only same/inconclusive, recall deltas null No production code changes.
…n semantics Clarify that Phase 2 is projection-only from existing shadow-comparison runs[], not a placeholder awaiting production baseline capture or applied simulation. Fixes: - delta-criteria.md: rewrite 'What Requires Counterfactual Projection (Phase 2)' to explain current Phase 2 IS the counterfactual projection from runs[]; remove stale claim that Phase 2 requires production baseline capture/recall side-by-side; update 'Before Phase 2' checklist to show items as checked (Phase 2 is already complete); rename section from 'Why delta.status=future Is Currently Correct' to 'Why delta.status=counterfactual Is the Correct Phase 2 Standing' - comparison-methodology.md: fix artifact example where supersede had projectedDelta=-1 (supersede should be 0; use reject for -1 example); clarify 'future' status is legacy/Phase-1 empty state (current impl only emits counterfactual); fix Quick Reference 'remains future' to 'remains null' for recall deltas - shadow-comparison.ts: update BalancedScore.delta comment from stale 'currently marked as future' to Phase 2 counterfactual projection description Conservative rule preserved: Phase 2 never emits better/worse, only same/inconclusive. Measured recall/memory quality impact and side-by-side recall belong to Phase 3.
Implements Phase 3 applied-simulation evaluation for the Memory Decision Engine using Path A design (storage lifecycle seams, not production active-mode wiring). New files: - src/benchmark/applied-simulation-evaluation.ts: Main harness with 14 scenarios covering current-truth/supersession, no-answer, rank-1 current-truth, cleaner-memory improvement, safety, invalidation, and no-op/store-agreement - src/benchmark/applied-simulation-evaluation-runner.ts: CLI runner with --artifacts, --no-write, --no-stdout flags - tests/applied-simulation-evaluation.test.ts: 24 tests covering module exports, hermeticity, report shape, artifact writer, human formatter, classification semantics, minimum scenario gate, and public API invariants Key features: - Uses storage lifecycle seams (transitionMemoryState, addSupersededByToMemory, updateMemoryMetadata) to apply smart-librarian decisions in cloned temp storage - Measures recall deltas (hit rate, current-truth rate, no-answer TNR, rank-1 match) against production baseline - Classification: better/same/worse/inconclusive with conservative rules - Minimum scenario gate: 14 scenarios satisfying coverage requirements - evaluationMode: 'applied-simulation', delta.status: 'applied' (measured, not counterfactual) - Allowed actions: store, no_op, supersede, invalidate, flag_redundant, flag_conflict, reject - Excluded from scoring: redact, clarify (type-constrained at build time) Safety constraints preserved: - No production DB/state access - No live network (scripted fetch only) - No MCP API changes - No active-mode engine wiring - actor: 'phase3-applied-simulation' tag on simulated transitions Updates benchmark README with Phase 3 documentation.
- Implement resolveGroundTruthLabels() for supersession/rank-1 scenarios:
- For categories 'current-truth-supersession' and 'rank-1-current-truth',
resolve currentTruthIds and expectedRank1Ids to the stored memory id
where honest (memory was actually stored by the applied path).
- For no-answer/safety/invalidation, labels remain as-is.
- Remove dead imports:
- DecisionAdapterSuccess (type not used in file)
- buildRecordingScriptedFetch (not used; okChatResponse still imported)
- Remove dead helper functions:
- runProductionBaseline() - never called, logic inlined in main function
- runAppliedSimulation() - never called, logic inlined in main function
- Use per-path ground truth with resolved labels in computeRecallMetrics:
- baselineRecall uses resolved baseline ground truth
- appliedRecall uses resolved applied ground truth
- Add tests proving:
- supersession scenarios have meaningful currentTruthAtRank1
- rank-1 scenarios have meaningful rank1Match when memory stored
- aggregate deltas are non-null when scenarios exist
- no-answer/safety scenarios don't incorrectly regress
- classifications not driven solely by memory-health when recall measurable
Verification:
- npx tsc --noEmit: pass
- npm run test:applied-simulation: 29 pass
- npm run test:shadow-comparison: 42 pass
- npm run benchmark:applied-simulation -- --no-write: pass
…r outcome Replace brittle substring lookup (memoryContent.includes(rawInput.slice(0,50))) with direct capture of outcome.record.id from the applied path runRememberController. This mirrors the baseline path behavior and fixes spurious 'worse' classification for paraphrase scenarios where provider summary differs from rawInput. In baseline path, we already capture outcome.record.id directly. Applied path was using a brittle listActiveMemorySummaries + find approach that failed when provider summary (utilizing) differed from rawInput (using). All 14 applied-simulation scenarios now pass (worse:0).
- Document Phase 2 as counterfactual projection only - Document Phase 3 as offline applied simulation in cloned temp storage - State current Phase 3 synthetic result: better: 8, same: 6, worse: 0 - Clarify better/same/worse are benchmark-internal synthetic classifications - Explicitly state activation remains blocked - Explicitly state recallHitRateDelta is null/missing - List next evidence gates: production replay corpus, shadow telemetry review process, rollback design, active-mode controller design, owner approval before any activation - No code behavior changes
- Phase 2 counterfactual projection emits same/inconclusive/null only; cannot emit better/worse (no decisions applied to store) - better:8, same:6, worse:0 correctly attributed to Phase 3 applied simulation in cloned temp storage - recallHitRateDelta note clarified: per-scenario Phase 3 classifications exist while aggregate field awaits population from full Phase 3 harness - Quick Reference rows corrected: Phase 2 benchmark result row renamed to Phase 3 applied-simulation result; recallHitRateDelta and Phase 2 description rows updated to match Phase 2 same/inconclusive/null only - Conservative framing preserved: synthetic/offline, not production readiness, activation remains blocked, next evidence gates listed
Add production-replay-corpus.md defining the corpus design and governance framework. Covers purpose, allowed sources (synthetic-proxy/manual-curated now; sanitised-production deferred until privacy approval), forbidden data, schema using stable content-derived IDs, privacy/redaction rules, ground-truth labelling process, minimum diversity targets, offline replay methodology, required metrics, artifact format, and approval gates. Cross-link added from promotion-readiness.md (Next Evidence Gates section) and comparison-methodology.md (Constraints section). Docs-only. No code changes. No corpus data. No replay harness. No evidence/ modification.
Removed '影子' token from line 326 in the No production mutation step. Clean English sentence, no code or behavior changes.
… noAnswerTnr non-null - Add 2 synthetic-proxy no-answer recall-query items (French greeting, geography question) with noAnswerExpected: true and empty expected labels - Clean stale comments in replay-corpus-evaluation.ts that said recall metrics were deferred to Phase 3 - Add 8 new tests for no-answer TNR: - noAnswerExpectedCount > 0 verification - noAnswerTnr non-null when no-answer items exist - noAnswerTnr correctness (1.0 in scripted replay) - no-answer items privacy validation - no-answer items have empty expected labels - no-answer items pass recall query privacy checks - repeated runs produce identical noAnswerTnr - no sanitised-production recall-query items exist - Update production-replay-corpus.md docs: - Update recall-quality note to reflect metrics are computed - Rename section from 'Deferred to Phase 3' to 'in Replay' - Update example JSON to show computed metrics (not nulls) - Update FAQ entry - All 54 replay-corpus tests pass - All 42 shadow-comparison tests pass - All 29 applied-simulation tests pass
- replay-corpus-evaluation.ts: update JSDoc to clarify recall quality metrics are computed from scripted engine responses and labeled ground truth; full/live retrieval quality remains out of scope - production-replay-corpus.md: update limitations statement to reflect replay recall metrics are offline scripted measurement infrastructure, not pending Phase 3
geanatz
marked this pull request as ready for review
July 10, 2026 00:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This draft PR prepares review and CI for the
experiment/adaptive-recall-depthbranch. It is experimental/readiness work, not an activation PR.Included workstreams:
Verification
Latest known full local test result for this HEAD:
npm test: 2210 total, 2198 passing, 0 failing, 12 skipped.CI did not run on the experiment branch directly because the workflow triggers on push to
mainor pull requests tomain. This draft PR is intended to trigger PR checks and support review.Explicit non-goals / activation blockers
origin/mainby one unreconciled commit:ee99005 chore(release): add MCP Registry metadata.Review focus
origin/maincommit before any merge path.