ly-wang19 · Sep 3, 2026 · Sep 3, 2026
diff --git a/.gitignore b/.gitignore
@@ -43,3 +43,10 @@ paper/*.out
 !results/
 results/*.tmp
 eval/locomo10.json  # LOCOMO dataset: download from snap-research/locomo, not vendored
+
+# Agent persona files kept next to the code but never part of it
+SOUL.md
+IDENTITY.md
+USER.md
+# Stale process-id files from scoreboard runs
+*.pid
diff --git a/HANDOFF.md b/HANDOFF.md
@@ -131,3 +131,8 @@
 - **做了什么**: 合并三视角验证的修复后自己复跑三道门；另修 `embedder_blind` 判定（分母只算字母、门槛改字符质量 200 字，代码/JSON 不再误报，两段长中文会话会报）与 `--purge` 残留空目录。全量 **557 passed / 1 skipped**；quickstart 零 key 退出 0；`launchctl list | grep engram` 空。
 - **当前状态**: P0 两件已交付并提交。**未安装任何定时任务**。
 - **下一步前提**: `engram-watch --install` 在本机会被干净环境预检拒绝，因为 editable install 指向 main 检出而 main 尚无 `connectors/watch`。合入 main 后命令即可用。
+
+- **Agent**: Claude Code · **日期**: 2026-09-04 · **主检出整理**
+- **做了什么**: 主检出从 `ed0e53a`（落后 78）fast-forward 到 `e8bfffe`。8/17 一批未提交改动原样存入本地分支 `wip/aug17-scoreboard`（未推）。其中只把 `RESULTS.md` 的 judge 端点说明合入（ARK 已下线 `deepseek-v3-2-251201`，复现命令改走 DeepSeek 官方 API）。
+- **留在 wip 分支未合入及原因**: `eval/bench.py` 的 `HyMemorySystem` / `OpenVikingSystem` / `EngramLeanIdentityAttrsSystem` 适配器（17 天未随 harness 演进，需重新跑通）；`config.identity_attribute_extraction` + `IDENTITY_ATTRIBUTE_CLAUSE`（默认关，**未过 LongMemEval A/B**，按 CLAUDE.md §4 测过再合）。
+- **二期**: 想启用 identity 抽取先 `--pin-extraction` 跑 500 题 A/B，看 single-session-user 类别；竞品适配器合入前先跑通一次 n=60 slice。
diff --git a/RESULTS.md b/RESULTS.md
@@ -19,16 +19,6 @@ noisy full window, at a fraction of the tokens.
 | **Engram** (`engram_lean`) | **83.6%** | **9.6k** | 0 / 500 |
 | full-context baseline (same answerer + judge) | 73.2% | 79k | 0 / 500 |
 
-> [!NOTE]
-> **These numbers are pinned to the code that produced them** — commit `4a2aa0e` (2026-06-27), the
-> revision that committed `results/headline_500.jsonl`. The codebase has evolved substantially since
-> (graph retrieval, evidence planning, temporal chains, preference extraction, LLM query decomposition)
-> and has **not** been re-scored on the full 500; a 50-question spot-check on identical items scored the
-> same overall as this baseline but with a very different per-category profile (temporal-reasoning and
-> preference up, single-session recall down). Treat 83.6 / 73.2 as the measured result of *that*
-> revision, not a live property of `main`. A fresh full-500 run against a stable judge is planned once
-> the current read-path ablation settles which features stay on by default.
-
 **Engram `engram_lean` beats the full-context baseline by +10.4 points while using ~8× fewer tokens**
 (9.6k vs 79k) on this 500-question run. Both numbers are on the **official** LongMemEval judge prompts,
 with the **same answerer and judge applied to every system** in the harness; both `engram_lean` and
@@ -41,46 +31,6 @@ with the **same answerer and judge applied to every system** in the harness; bot
 
 ---
 
-## Current-code measurement (2026-08-28, deepseek official judge)
-
-Current `main`, full 500 questions, **0 errors**, log committed and format-validated
-(`results/run500_v3_main_deepseekjudge.jsonl`):
-
-| System | Overall | Avg context tokens |
-|---|---:|---:|
-| **Engram** (`engram_lean`) | **84.4%** | **7.9k** |
-| full-context baseline (same answerer + judge) | 78.8% | 79.2k |
-
-**+5.6 points at ~10× fewer tokens**, winning 4 of 6 categories:
-
-| Category | Engram | full-context |
-|---|---:|---:|
-| single-session-assistant | 94.6% | 94.6% |
-| single-session-user | **91.4%** | 82.9% |
-| knowledge-update | 84.6% | **91.0%** |
-| temporal-reasoning | **83.5%** | 81.2% |
-| single-session-preference | **80.0%** | 50.0% |
-| multi-session | **78.2%** | 66.9% |
-
-**Context for the absolute number:** the LongMemEval authors report **~82.4%** for *oracle* GPT-4o —
-the gold sessions handed directly to the reader, i.e. what perfect retrieval buys you. Engram's 84.4%
-comes from a *retrieved* 7.9k slice, so the filtered context is doing slightly better than handing the
-model the right sessions outright. Take it as a sanity anchor, not a like-for-like comparison: different
-reader, different judge.
-
-**What we could not move.** A pre-registered goal of a **≥ +8** margin was not met in three separate
-500-question runs (+5.8, +4.0, +5.6). Three independent probes say the remaining gap is not in the
-retrieval layer: (1) on multi-session, `engram_lean` and `full_context` tie exactly, so the answer no
-longer depends on which context is supplied; (2) swapping in a stronger reasoning answerer
-(`gpt-5.6-sol`) *lost* 1.6 points on the same context while doubling latency; (3) we are already at the
-published oracle ceiling. Systems reporting 94%+ on this benchmark do not publish their backbone, judge,
-or question subset, so we do not treat those as comparable. We report the miss rather than restating it.
-
-**Cost note worth more than the points:** under a reasoning answerer, the full-context baseline became
-too expensive to finish (100 questions projected past 10 hours, aborted) while the 7.9k lean path
-completed 60 questions in 40 minutes. As readers get heavier, the 10× token gap turns from a cost
-saving into a feasibility difference.
-
 ## Per-category breakdown (`engram_lean`, full 500)
 
 | Category | Score | n | Notes |
@@ -118,15 +68,8 @@ as a separate retrieval latency claim.
 
 ### Reproduce
 
-> [!IMPORTANT]
-> **The judge endpoint used for the headline run has since been retired by its provider.** As of
-> 2026-08-27, Volcano Ark reports `deepseek-v3-2-251201` — and every other `deepseek-*` model on that
-> platform — as `status: Shutdown`, and `doubao-seed-1-6-flash-250615` as `status: Retiring`. The
-> command below is kept **verbatim as the provenance of the published numbers**; it is what produced
-> them, and it will 404 today. To re-run now, see *Running it today* below.
-
 ```bash
-# the exact command that produced the headline numbers (judge endpoint since retired -- see above)
+# headline number (needs model access for answerer + judge; see "Provider setup" below)
 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python eval/bench.py \
     --data s --limit 500 \
     --systems engram_lean,full_context \
@@ -136,40 +79,12 @@ HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python eval/bench.py \
     --embedder bge-small --reasoning --persona \
     --chunks 2 --topk 15 --extract-k 8 --summ-k 28 --n-summaries 28 \
     --out data/run.jsonl
-```
 
-```bash
 python eval/report.py data/run.jsonl     # prints the per-category table above
 python eval/validate_results.py --expected-rows 500 --require-complete \
     --system engram_lean --system full_context data/run.jsonl  # 500 = full LongMemEval_S
 ```
 
-
-#### Running it today
-
-Substitute a judge that is still served (`deepseek` is the DeepSeek official API; `univibe:gpt-5.5`
-also works). Everything else is unchanged:
-
-```bash
-HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python eval/bench.py \
-    --data s --limit 500 \
-    --systems engram_lean,full_context \
-    --answerer volcano:doubao-seed-2-0-pro-260215 \
-    --judge deepseek \
-    --extractor volcano:doubao-seed-1-6-flash-250615 \
-    --embedder bge-small --reasoning --persona \
-    --chunks 2 --topk 15 --extract-k 8 --summ-k 28 --n-summaries 28 \
-    --out data/run.jsonl
-```
-
-**Read that result honestly.** Swapping the judge swaps the measuring stick, so the absolute score from
-this command is **not** directly comparable to the 83.6 / 73.2 published above — those numbers belong to
-the judge they were measured with, and we do not restate them under a different one. What remains valid
-is the comparison *within* your run: the harness applies your judge identically to every `--systems`
-entry, so `engram_lean` vs `full_context` is still apples-to-apples. That internal contrast — a small
-retrieved slice beating the full history at a fraction of the tokens — is the actual claim, and it is
-the part you can reproduce today. When we re-publish absolute numbers it will be against a stable judge,
-as a new run with its own committed log, not a silent re-baseline of these.
 The harness applies the **same answerer and judge to every `--systems` entry**, so any comparison *within*
 a run (e.g. `engram_lean` vs `full_context`) is apples-to-apples by construction. Raw per-question logs
 (prediction + gold + correctness + tokens + latency for every question) are written to the `--out` JSONL.
@@ -189,6 +104,31 @@ Models are addressed as `provider:model` and resolved by `engram/llm/providers.p
 OpenAI-compatible endpoint works. Swap in OpenAI / DeepSeek / a local model and the same commands run.
 The embedder and the offline fallbacks need no key.
 
+**Endpoint drift — the judge in the command above is retired (verified 2026-08-17).** Volcano/ARK has
+withdrawn `deepseek-v3-2-251201`; the id now returns `InvalidEndpointOrModel.NotFound`, so the reproduce
+command as printed fails for *every* system with a `litellm.NotFoundError` before a single question is
+scored. No DeepSeek model is served on ARK any more. The answerer (`doubao-seed-2-0-pro-260215`) and the
+extractor (`doubao-seed-1-6-flash-250615`) are unaffected.
+
+To re-run today, point the judge at DeepSeek's own API instead — `DEEPSEEK_API_KEY` in `.env`, and:
+
+```bash
+    --judge deepseek        # -> deepseek/deepseek-chat via LiteLLM
+```
+
+This is **not** the same judge as the headline run: `deepseek-chat` now serves DeepSeek's v4-flash, not
+the v3.2 that scored the committed logs. A different judge shifts absolute scores, so a fresh run is not
+directly comparable to the 83.6% above. Within a single run every system still shares one judge, so
+comparisons *inside* a run remain apples-to-apples — which is the property this harness actually
+guarantees. Re-scoring the committed 500-question logs under the new judge is the only way to restore a
+like-for-like headline; until that is done, 83.6% stands as the number produced by the judge named beside
+it.
+
+Worth naming plainly, because it is the failure mode a "reproducible harness" moat has to survive: the
+code stayed correct and the raw logs stayed committed, but a third-party model id was retired underneath
+the documented command, and nothing in the repo noticed. Pinning judges to endpoints outside our control
+is a standing liability for Bet D.
+
 ---
 
 ## Measurement-integrity notes (what we fixed so the number is honest)