Skip to content

test(recall): cache accounting benchmark for split_system (SIMULATION)#450

Closed
muzimu217 wants to merge 1 commit into
TencentCloud:mainfrom
muzimu217:fix/split-system-benchmark
Closed

test(recall): cache accounting benchmark for split_system (SIMULATION)#450
muzimu217 wants to merge 1 commit into
TencentCloud:mainfrom
muzimu217:fix/split-system-benchmark

Conversation

@muzimu217

Copy link
Copy Markdown

摘要

#449split_system 把稳定 persona 前置到 CACHE_BOUNDARY)补充一个缓存记账基准测试,验证 split_system 确实增大了"缓存边界之前"的可缓存前缀长度。

⚠️ 本文档/测试是确定性仿真,不是真实 A/B。
它不调用任何 LLM API,不测量真实 cache_read / cache_creation token。
真实测量方法请见社区 PR #433(2 provider × 2 variant × 2 repeat × 3 turn = 24 个实测 cell)。本 PR 只做两件事:① 文档化 split_system 的预期行为;② 防止前缀组装回归。

为什么需要这个测试

前缀匹配缓存(DeepSeek / MiMo / Anthropic 类)只对 CACHE_BOUNDARY 之前、且逐 token 完全一致的前缀做缓存。split_system 的价值 = 把唯一稳定的 persona 移进这段可缓存区域。本测试断言的唯一不变量:

  • split_system 开启时,缓存前缀长度 严格大于 none
  • persona(稳定内容)在边界前;动态 L1 记忆(每轮变化)不在边界前。

测试内容

src/adapters/openclaw/cache-accounting.benchmark.test.ts(4 用例):

  1. split_system 缓存前缀含 persona 且长于 none
  2. 每轮 cache_read 增益可观且确定性(约 10–40 token)
  3. 20 轮会话:split_system 累计 cache_read 严格更大,且增益随轮数线性(无随机)
  4. persona 是唯一移入边界前的内容;动态 L1 标记(<relevant-memories> / <memory-context>)不应出现在边界前

#449 / #375 / #433 的关系

Scope / Non-scope

Deterministic simulation that verifies split_system increases the cacheable
prefix (before CACHE_BOUNDARY) by moving the stable persona into it.

Clearly labeled as SIMULATION — it does NOT call any LLM API and does NOT
measure real cache_read/cache_creation tokens. For real measured numbers,
see community PR TencentCloud#433 (2 providers x 2 variants x 2 repeats x 3 turns).

The single invariant asserted: with split_system, the cached prefix is
strictly longer than without it, because the persona (only stable recall
artifact) is moved before the boundary. Dynamic L1 memories stay after.

This guards against regressions in how split_system assembles the prefix;
it is a documentation + regression check, not a performance claim.
@YOMXXX

YOMXXX commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Duplicate/scope cleanup triage:

This is useful coverage, but it should not stay as a standalone PR. The test file only validates the split_system behavior introduced by #449, and the PR body explicitly treats #449 as the implementation under test. If #449 is accepted, this benchmark/regression test should be folded into #449 so the behavior and its guard land together. If #449 is not accepted, this PR has no independent production behavior to protect.

I would close this as a separate PR and move the test into #449 during the next revision. The real-provider measurement lane should remain separate from this simulation, as noted in the PR body.

@muzimu217

Copy link
Copy Markdown
Author

This test suite has been superseded by improved tests in #449.

Issues with this PR:

  • Tests were tautological (testing own mock)
  • Did not verify real host behavior
  • No value as standalone PR

Replacement:

Closing to consolidate testing in the main PR.

@muzimu217 muzimu217 closed this Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants