Repository navigation
Commit debd3c7
authored
feat(settings): user-facing prompt-cache TTL for Anthropic profiles (C6) (#2271)
* fix(openai-native): send body prompt_cache_key for cache affinity
What: the OpenAI-native handler now sends a task-scoped `prompt_cache_key`
in the Responses request body on both body builders (streaming and
completePrompt), and no longer sends the `session_id` request header.
The key is `metadata.taskId`, falling back to the handler session id,
clamped to OpenAI's 64-character limit, and omitted for models whose
catalog entry has `supportsPromptCache: false`.
Why: OpenAI keys prompt-cache routing off the body `prompt_cache_key`
field. The `session_id` header we were sending is not part of the
prompt-caching API and is ignored on the public Responses API, so every
OpenAI-native task was leaving cache hit-rate on the table. The header
also carries an underscore in its name, which hardened gateways
(Envoy/nginx `underscores_in_headers`) reject outright, so it was a
latent 400 for anyone behind such a proxy with a custom base URL.
`prompt_cache_retention: "24h"` is untouched: it is the supported
lifetime control on the pre-5.6 models that set it in the catalog, and
the existing retention specs pass unmodified.
Impact: cache affinity now follows the documented body field, stable
across turns of one task and distinct across tasks. No usage
accounting, cost, or other request field changes. openai-codex.ts and
xai.ts are untouched: codex already sends a body key, and xai never had
the header.
* fix(cost): account cache writes per TTL class instead of one aggregate rate
What:
- Add `cacheWritesPrice1h` / `cacheWritesPrice12h` to the ModelInfo schema plus
`deriveCacheWritesPrice1h` / `deriveCacheWritesPrice12h`, so the vendor
multiplier relationship is stated once in model.ts.
- Bill three write classes in `calculateApiCostInternal`: the 5m remainder is
derived as total - 1h - 12h (clamped at 0), so the classes always sum to the
aggregate and no token is billed twice or dropped.
- Scale the TTL rates with `cacheWritesPriceMultiplier` in long-context pricing.
- Carry `cacheWrite1hTokens` / `cacheWrite12hTokens` on `ApiStreamUsageChunk` and
read them from Anthropic `usage.cache_creation.ephemeral_1h_input_tokens`
(anthropic, minimax, anthropic-vertex) and OpenAI Responses
`input_tokens_details.cache_write_tokens_details.ttl_12h_tokens`
(openai-native `normalizeUsage` only).
- Thread the split through the Task usage accumulator, including the cancel-path
cost recomputation that runs when a provider yields no totalCost.
- Populate the rates: 22 Anthropic entries, 19 Bedrock Anthropic-family entries,
7 OpenAI GPT-5.6/6-class entries.
Why:
Anthropic bills a 1h cache write at 2x input versus 1.25x for 5m, and OpenAI bills
a 12h write at 2.5x versus 1.25x for 30m. The repo read only the aggregate
`cache_creation_input_tokens` and priced every write at the short rate, so a 1h
write under-reported by 37.5% and a 12h write by 50%. The vendors already report
the per-TTL split; it was being discarded.
Impact:
Reporting only. No request shape changes: the repo emits no `ttl` field, so live
traffic still reports 0/absent long-lived writes and pricing is byte-identical to
before (asserted against the historical cost literals). When the TTL knob lands,
a 1h write can no longer be billed at the 5m rate. 18 new tests fail on the base
blob; the new cost code has no surviving mutants.
* fix(cost): bill the TTL split on the telemetry recompute and in tier blocks
What:
- Pass `cacheWrite1h`/`cacheWrite12h` to the cost call inside `captureUsageData`.
The bag already carried them and fed the shared accumulator vars, but the
telemetry recompute passed only the aggregate write count, so a cancelled
request's telemetry billed every write at the short-TTL rate.
- Add `cacheWritesPrice1h`/`cacheWritesPrice12h` to the `tiers` schema and to the
four Anthropic and four Bedrock 1M-context tier blocks, derived from each
tier's own input price.
- Carry the tier's TTL rates through all three tier-merge seams (anthropic,
anthropic-vertex, bedrock) and derive them from the tier input price in the
OpenAI service-tier seam, so a flex/priority or 1M-context write is not billed
at the base tier's TTL rate.
Why:
Two under-reports survived the first pass. `captureUsageData` recomputes cost for
telemetry when a provider yields no final cost chunk (the cancelled-stream path),
and it dropped the split: 100k 1h tokens at claude-3-5-sonnet billed 0.375
instead of 0.6. Separately, the 1M-context tier merge overrode `inputPrice` but
not the TTL rate, so the entry-level rate derived from the 200K input price
survived: a 1M-context 1h write billed 6.0 instead of the tier's 12.0, a 50%
under-report.
Impact:
Both paths now bill at the correct TTL rate. Three new tests fail on the
pre-fix code (0.375 vs 0.6, and 6 vs 12); a fourth asserts the tier rate reaches
the cost function. No request shape changes.
* test(cost): correct the mixed-split test label to match its arithmetic
What:
- Rename "bills a mixed 5m/1h split per class (0.465 case)" to
"(60/40 split, 0.51)".
Why:
The label cited a figure the test does not assert. A 60/40 split of 100k writes
at 3.75/6.0 costs 0.15 + 0.36 = 0.51, not 0.465. The assertion was already
correct; only the name was misleading, and a reviewer reading the label would
have expected a different number.
Impact:
Test name only, no behaviour change.
* test(openai-native): cover the 12h rate under each service tier
What:
- Add a per-tier case asserting the 12h write rate follows the tier's own input
price (default 25.0, flex 12.5, priority 50.0 for gpt-6-astra).
Why:
The service-tier seam was changed to derive the TTL rates from the tier input
price, but nothing pinned that behaviour. Without a test, a future edit could
revert to the standard-tier 12h rate and a flex/priority 12h write would be
mis-billed by 2x in either direction.
Impact:
Test only. Fails without the tier derivation (flex bills 2.5 instead of 1.25,
priority 2.5 instead of 5.0).
* feat(api): emit an opt-in prompt cache TTL gated by catalog capability
What: add a `promptCacheTtl` provider-settings knob ("5m" | "1h"), a
`ModelInfo.promptCacheTtlSupport` capability array, and a shared
`resolveCacheTtl`/`cacheControl` helper that the anthropic, vertex and
vercel-ai-gateway caching transforms plus the anthropic, anthropic-vertex,
openrouter, vercel-ai-gateway and zoo-gateway providers thread through.
Why: ADR-0006 phase 2. C5a (#2260) landed per-TTL write accounting, so a 1h
cache write is billed at 2x instead of the 5m rate. This is the matching
request-side mechanism: once the settings UI ships in C6, a user selecting 1h
causes Anthropic-shaped providers to emit `cache_control: { ttl: "1h" }`.
Impact: no behavior change by default. An unset knob resolves to `undefined`
and the marker stays the byte-identical `{ type: "ephemeral" }` literal, so no
`ttl` key is emitted unless the user opts in AND the resolved ModelInfo
advertises the capability. Absent capability means 5m only (I2), capability is
read from ModelInfo rather than id substrings, and every marker in a request
shares one resolved TTL (I4). Dynamic catalogs (OpenRouter, the gateways) carry
no capability data and therefore stay at 5m, as the ADR requires.
* feat(settings): offer the prompt cache TTL control on Anthropic profiles
What: adds the 1h cache TTL selector to the Anthropic provider panel, gated
on the model's catalog capability, plus the four locale keys in all 18
locales.
Why: ADR-0006 decision item 3 ships the knob last, after the accounting
(C5a) and the mechanism (C5b) so no request can emit a 1h write that is
not priced. The control is the user-facing half; it renders only where
promptCacheTtlSupport advertises 1h (invariant I2), and selecting the
default writes DEFAULT_PROMPT_CACHE_TTL (undefined) rather than a literal
"5m" so an unset knob keeps emitting no ttl key (invariant I3).
Impact: Anthropic profiles gain an opt-in control; default behavior and
wire bytes are unchanged for every profile that leaves it alone.
Test: webview-ui Anthropic.spec.tsx 8/8.
* test(settings): cover the prompt cache TTL settings round trip
What: adds the round-trip coverage the Persisted Setting Checklist asks for:
extension-side survival through getStateToPostToWebview (set and unset),
import/export schema preservation of both "1h" and "5m", and the
SettingsView cachedState buffer (restore, buffer-until-Save, and the
absence-vs-explicit-5m distinction).
Why: the knob rides apiConfiguration, so a missing schema field or a
missing state-post field would silently revert the control without any
type error. The import/export case is falsified by removing the schema
entry (test goes red), so it guards the field rather than the fixture.
Impact: test-only; no production behavior change.
Test: src 504/504 across the six affected suites, webview-ui 2235/2235.
* fix(api,cost): hash long cache keys, clamp TTL splits, harden the TTL specs
What:
- openai-native: hash over-long task ids with sha256 hex instead of slicing to
64 chars. A slice drops the distinguishing tail, so two ids sharing a 64-char
prefix collapsed into one routing bucket. sha256 hex is exactly 64 chars, so
the result still fits OpenAI's cap.
- openai-native: clamp the 12h write subset at zero as well as at the write
total. A negative `ttl_12h_tokens` from a misbehaving gateway used to reach
the Task accumulator and the webview as a negative count.
- cost: cap the 1h and 12h write subsets at the remaining aggregate write total.
A malformed split (1h + 12h > total) billed the excess; 150k reported 1h
writes against a 100k total cost 0.9 instead of 0.6.
- Task.spec: replace the fixed 50ms sleep with `vi.waitFor` on the captured
telemetry call. The drain is fire-and-forget and touches the filesystem, so a
fixed sleep races it on a loaded worker.
- vercel-ai-gateway.spec: assert the TTL argument of the `addCacheBreakpoints`
mock call. The transform is file-mocked, so `JSON.stringify(mockCreate)` can
never contain `cache_control` and the old assertion could not fail.
- ttl.spec: compare raw `JSON.stringify` output against literal expected strings
in the byte-identical test. `toEqual` on round-tripped objects ignores key
insertion order, so a key reorder passed while the wire bytes changed.
Why:
Seven CodeRabbit "Minor" threads on the C5a/C5b stack. Three are production
correctness (cache-key collision, negative token count, overbilled split); three
are assertions that could not fail. The seventh thread's premise, forwarding the
TTL split into the telemetry recompute, already landed in b647ec7.
Impact:
Long task ids keep distinct cache buckets; malformed TTL splits can no longer
over- or under-bill or leak negative counts. No request shape changes for the
default path: the aggregate-only case still derives both subsets as zero.
* test(webview): mount the Anthropic settings form in the visual gallery
What: adds an "anthropic-settings" gallery story and a matching visual test
that asserts the prompt cache TTL control's copy, default value, sibling
alignment, contrast and overflow behavior.
Why: the C6 control is a user-visible settings surface, and the repo asks
for gallery coverage on changes a reviewer would notice at a glance. The
story pins the default Anthropic model (claude-sonnet-4-5), whose catalog
entry advertises promptCacheTtlSupport ["5m","1h"], so the capability-gated
control actually renders.
Deliberately no toHaveScreenshot baseline: committed baselines are rendered
in the CI Linux container (visual-regression.yml) and run-docker.mjs states
host-rendered baselines are not reachable, so a Windows-rendered PNG would
fail the required webview-visual check. The assertions are geometric and
textual instead, following the LayoutContracts.visual.tsx precedent, and
they are falsifiable (a 7px injected offset turns the test red).
Impact: test/visual-infra only; no production behavior change.
Test: playwright -c playwright-ct.config.ts Anthropic.visual.tsx 1/1,
stable across two consecutive runs; eslint and tsc clean.
* fix(api,core): key prompt enhancement off the task id, not the handler session
`completePrompt` built its `prompt_cache_key` from `this.sessionId`, which is
regenerated for every handler instance. Prompt enhancement builds a fresh
handler per request (`singleCompletionHandler`), so repeated enhancements of
one task each started a new cache bucket and could not reuse the previous
prefix.
`CompletePromptOptions` now carries an optional `taskId`. `MessageEnhancer`
sets it from the open task, `webviewMessageHandler` passes
`currentCline?.taskId`, and the OpenAI native handler feeds it through the
existing `getPromptCacheKey`, which already prefers `metadata.taskId` and
falls back to the session id. Callers with no task keep the old behaviour.
Tests: two new provider cases (task-scoped key across handler instances,
session fallback without a task id) and two enhancer cases (task id
forwarded, options omitted when no task is open). The provider case fails on
the pre-fix code with the session uuid in place of the task id.
* fix(settings): associate the prompt cache TTL label and pin the control
What: type the six Anthropic spec mocks against the real control props,
give the prompt cache TTL label an htmlFor bound to the trigger id, and add
a label-activation assertion plus a pixel baseline to the visual test.
Why: CodeRabbit review of PR #2271 raised three Minor findings. The untyped
mocks let a wrong prop shape pass unnoticed, and the label wrapped no control,
so clicking it did nothing for pointer users.
Impact: label clicks now open the selector; the spec fails on a wrong-typed
callback, and the visual test fails when the label association is removed.
* test(webview): commit the container-rendered prompt cache TTL baseline
What: add the anthropic-prompt-cache-ttl-dark.png baseline rendered by the
pinned Playwright container in the webview-visual CI job.
Why: webview-ui/AGENTS.md forbids creating committed baselines from host
rendering, and Docker is unavailable on the authoring host, so the CI
container fallback supplies the authoritative PNG.
Impact: the visual test's screenshot assertion now has a baseline that
matches CI, closing the last CodeRabbit finding on PR #2271.
* test(api): cover the accepted prompt cache TTL on the gateway call site
What: add a dedicated `anthropic/claude-3.5-sonnet` catalog row carrying
`promptCacheTtlSupport: ["1h"]`, and a test asserting the third argument to
`addCacheBreakpoints` resolves to `"1h"` when the knob is set.
Why: the existing pair only proved the reject path (`lastCall[2]` undefined)
and a bare `toHaveBeenCalled`. Nothing proved the accept path, so a regression
that dropped the third argument at the call site stayed invisible. The real
gateway catalog never sets `promptCacheTtlSupport`, so the capability has to be
injected by the test catalog rather than assumed from a real model.
Impact: test-only. The shared haiku row keeps no TTL capability, so the
existing reject-path test still asserts fail-closed behavior.1 parent a4f774b commit debd3c7
63 files changed
Lines changed: 2765 additions & 44 deletions
File tree
- packages/types/src
- __tests__
- providers
- src
- api
- providers
- __tests__
- transform
- caching
- __tests__
- core
- config/__tests__
- task
- __tests__
- webview
- __tests__
- utils/__tests__
- webview-ui
- playwright/gallery
- src
- components/settings
- __tests__
- providers
- __tests__
- __screenshots__
- i18n/locales
- ca
- de
- en
- es
- fr
- hi
- id
- it
- ja
- ko
- nl
- pl
- pt-BR
- ru
- tr
- vi
- zh-CN
- zh-TW
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
122 | 122 | | |
123 | 123 | | |
124 | 124 | | |
| 125 | + | |
125 | 126 | | |
126 | 127 | | |
127 | 128 | | |
| |||
166 | 167 | | |
167 | 168 | | |
168 | 169 | | |
| 170 | + | |
169 | 171 | | |
170 | 172 | | |
171 | 173 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
95 | 95 | | |
96 | 96 | | |
97 | 97 | | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
98 | 112 | | |
99 | 113 | | |
100 | 114 | | |
| |||
112 | 126 | | |
113 | 127 | | |
114 | 128 | | |
| 129 | + | |
| 130 | + | |
115 | 131 | | |
116 | 132 | | |
117 | 133 | | |
| |||
136 | 152 | | |
137 | 153 | | |
138 | 154 | | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
139 | 162 | | |
140 | 163 | | |
141 | 164 | | |
| |||
180 | 203 | | |
181 | 204 | | |
182 | 205 | | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
183 | 211 | | |
184 | 212 | | |
185 | 213 | | |
| |||
188 | 216 | | |
189 | 217 | | |
190 | 218 | | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
191 | 247 | | |
192 | 248 | | |
193 | 249 | | |
| |||
0 commit comments