translate: send session-affinity hints to openai_gateway and anthropic_gateway endpoints by devin-ai-integration[bot] · Pull Request #1101 · workweave/router

devin-ai-integration · 2026-08-28T18:10:35Z

Summary

applySessionAffinity deliberately sent no cache-affinity hint to gateway providers ("no affinity contract; may reject unknown headers"). In a customer's 2026-08-27 benchmark run this meant grok-4.6 served via their BYOK openai_gateway fanned across replicas with ~11% cache-read rate: 53.3M full-price input tokens vs 6.5M cache reads, inflating cost and pushing uncached 30k-token prefills toward the 30s ResponseHeaderTimeout (189 status-0 failures; see WorkWeave docs/eval/RESULTS_snowflake_clone_sqlbench_roster163a_20260827.md Follow-up 10).

Fix avoids the unknown-header/unknown-field concern by riding spec fields the gateway forwards with the body:

openai_gateway joins the existing ProviderOpenAI case: prompt_cache_key = session key, else stable prefix hash, else unhinted; caller-supplied key preserved.
anthropic_gateway gets a new applyAnthropicSessionAffinity on the PrepareAnthropic path: metadata.user_id = session key, only when the caller didn't send one (Claude Code's own bundle is already a usable affinity key). First-party Anthropic is untouched. Cortex rejects non-spec top-level keys, so a spec Messages field is the safe carrier.

Coverage map pins ProviderOpenAIGateway: mechanismPromptCacheKeyBody, plus dedicated unit tests for both gateways (set / preserve-caller / direct-provider-unhinted / empty-no-op).

wv router tc and wv router t pass.

Link to Devin session: https://app.devin.ai/sessions/85739dd5156f4ffd9b4ce1941cd91e13
Open in Devin Desktop: https://app.devin.ai/desktop/session/85739dd5156f4ffd9b4ce1941cd91e13?variant=devin
Requested by: @steventohme

…oints

greptile-apps · 2026-08-28T18:10:40Z

PR author is not in the allowed authors list.

devin-ai-integration · 2026-08-28T18:10:40Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment, CI, and merge conflict monitoring

claude · 2026-08-28T18:10:57Z

Claude finished @devin-ai-integration[bot]'s task —— View job

✓

…ndpoints

claude · 2026-08-28T18:15:29Z

Claude finished @devin-ai-integration[bot]'s task —— View job

Run git diff to analyze added comments
Post comment-length nit review

Posted one advisory suggestion on internal/translate/emit_anthropic.go lines 49–58: the 10-line godoc for applyAnthropicSessionAffinity can be cut to 4 lines without losing any of the WHY.

workweave-bot

Advisory only — comment-length nit. Won't block merge.

claude · 2026-08-28T18:18:01Z

Claude finished @devin-ai-integration[bot]'s task —— View job

Run git diff to see added comments
Analyze comment blocks for verbosity
Post review or exit silently

No comment blocks of 3+ lines warranting a flag. The applyAnthropicSessionAffinity godoc (4 lines) was already trimmed per the previous suggestion. Other added comment lines are either 2-line blocks or inline modifications to existing comments.

View job

devin-ai-integration · 2026-08-28T18:42:26Z

Runtime Cortex-style gateway verification

Verified the router locally against a strict bearer-authenticated Cortex-style mock gateway (rejects unknown top-level body fields with 400).

✅ anthropic_gateway: two multi-turn /v1/messages turns returned 200 with identical generated metadata.user_id = e10bf688971fe1e2c519628a0b504373.

✅ Caller Anthropic metadata: metadata.user_id: caller-session-xyz preserved exactly.

✅ Claude Code: claude -p completed end-to-end through router + mock gateway (MOCK-CORTEX-OK); Claude Code's own metadata.user_id preserved, not clobbered.

✅ openai_gateway: two /v1/chat/completions turns returned 200 with stable prompt_cache_key=873f88cb60f562b3a32432c94832d35b; no metadata injected.

⚠️ Caller-supplied prompt_cache_key: preserved when no session key exists (verified: caller-pck-1 passed through), but overwritten by the affinity key when a session key exists. This matches the code's fallback-branch comment but is asymmetric with the Anthropic path, where the caller's metadata.user_id always wins — flagging for an intent check.

✅ Regression, direct first-party anthropic: two-turn session routed with decision_provider=anthropic, auth via x-api-key, and no metadata/metadata.user_id injected — first-party bodies unchanged.

✅ Strict-gateway compatibility: every gateway-bound body passed the strict field allowlist (all 200s).

… 400

…pletions ingress

Resolve internal/proxy/service.go: keep main's surface-based dispatchOpenAI refactor and re-apply the prompt_cache_key strip-and-retry (per-attempt StripPromptCacheKey, pre-commit retry, endpoint memoization) on top.

claude · 2026-08-28T19:27:01Z

Claude finished @devin-ai-integration[bot]'s task —— View job

Run git diff to analyze added comments
Analyze comment blocks for verbosity
Post review or exit silently

Posted one advisory suggestion on internal/proxy/prompt_cache_key_retry_test.go lines 42–45: the 4-line test function godoc can be cut to 3 lines without losing the WHY.

View job

workweave-bot

Advisory only — comment-length nit. Won't block merge.

claude · 2026-08-28T19:29:41Z

Claude finished @devin-ai-integration[bot]'s task —— View job

Run git diff to analyze added comments
Analyze comment blocks for verbosity
Post review or exit silently

Posted one advisory suggestion review with 3 nits on comment-length:

internal/proxy/prompt_cache_key_retry_test.go L19–21: pckStrictGatewayProvider godoc restates the type name and Proxy method; contract fits in 2 lines.
internal/translate/envelope.go L46–49: StripPromptCacheKey field comment is 4 lines; the WHY (trailing-schema rejection) fits in 3.
internal/translate/session_affinity_test.go L352–354: 3-line test comment restates the function name; the invariant fits in 1.

workweave-bot

Advisory only — comment-length nits. Won't block merge.

claude · 2026-08-28T19:33:34Z

Claude finished @devin-ai-integration[bot]'s task —— View job

Run git diff to analyze added comments
Analyze comment blocks for verbosity
Post review or exit silently

Posted one advisory suggestion review with 3 nits on comment-length:

internal/providers/provider.go L516–519: IsUpstreamPromptCacheKeyRejection godoc is 4 lines; the WHY (trailing-schema gateways, retry license) fits in 3.
internal/providers/provider_retry_test.go L266–268: TestIsUpstreamPromptCacheKeyRejection godoc is 3 lines; the invariant fits in 2.
internal/proxy/service.go L3442–3445: inline 4-line comment before the strip-and-retry block; the WHY fits in 3.

View job

workweave-bot

Advisory only — comment-length nits. Won't block merge.

claude · 2026-08-28T19:37:52Z

Claude finished @devin-ai-integration[bot]'s task —— View job

Run git diff to analyze added comments
Analyze comment blocks for verbosity
Post review or exit silently

Posted one advisory suggestion review with 2 nits on comment-length:

internal/proxy/service.go L5918–5920: 3-line cross-reference before the prompt_cache_key strip-and-retry block in ProxyOpenAIChatCompletion; fits in 2.
internal/translate/session_affinity_coverage_test.go L40–42: 3-line map-entry comment explaining why openai_gateway uses prompt_cache_key; fits in 2.

View job

workweave-bot

Advisory only — comment-length nits. Won't block merge.

workweave-bot

Advisory only — comment-length nits. Won't block merge.

claude · 2026-08-28T19:41:21Z

Claude finished @devin-ai-integration[bot]'s task —— View job

Run git diff to analyze added comments
Analyze comment blocks for verbosity
Post review or exit silently

No comment blocks of 3+ lines warranting a flag. All added blocks that reached 3+ lines were already reviewed and trimmed in this PR's previous commits (a79ab211, 913bd668, a4fdc294, 7f260257). Remaining blocks explain genuinely non-obvious behavior (memoization semantics, spec-field safety, retry licensing) concisely.

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a79ab21. Configure here.}

claude · 2026-08-28T19:59:09Z

Claude finished @devin-ai-integration[bot]'s task —— View job

Run git diff to analyze added comments
Analyze comment blocks for verbosity
Post review or exit silently

Posted one advisory suggestion review with 2 nits on comment-length:

internal/translate/emit_openai_responses.go L48–52: applyResponsesSessionAffinity godoc is 5 lines; the WHY (spec field forwarding, reasoning-tool promotion) fits in 3.
internal/translate/session_affinity_test.go L387–389: 3-line test comment before TestSessionAffinity_ResponsesCarriesPromptCacheKey; the invariant fits in 2.

devin-ai-integration · 2026-08-28T19:59:18Z

Runtime verification vs Cortex-shaped mock gateways (both surfaces)

Ran the branch build locally against two mock gateways: a Cortex-faithful one (accepts prompt_cache_key / metadata.user_id) and a strict spec-trailing one (400s prompt_cache_key as "unknown field").

openai_gateway: per-session stable prompt_cache_key (e50b2bf1… on both turns of one session), distinct key for a different session (1767cdc8…); all 200 on /v1/chat/completions.
anthropic_gateway: stable injected metadata.user_id (6e979565… across turns), distinct per session, and a caller-supplied metadata.user_id (caller-session-xyz) passed through untouched.
Accepting (Cortex-like) gateway: exactly one dispatch per turn — no spurious retries.
Strict gateway strip-and-retry: turn 1 → 400 carrying the key → one pre-commit retry without it → client 200; turn 2 → single request with no key (endpoint memoized, no second 400); exactly one router warn log.

One gap found: the Responses emit path (PrepareOpenAIResponses) carried no prompt_cache_key, so gateway reasoning-tool turns that promote to /v1/responses (the real Cortex endpoint serves it and accepts the field) went out unhinted. Closed in d2112c8: the Responses body now gets the same per-session key / caller-key preservation / prefix-hash fallback / StripPromptCacheKey semantics as chat/completions.

workweave-bot

Advisory only — comment-length nits. Won't block merge.

claude · 2026-08-28T20:02:02Z

Claude finished @devin-ai-integration[bot]'s task —— View job

Run git diff to analyze added comments
Analyze comment blocks for verbosity
Post review or exit silently

Posted one advisory suggestion review with 2 nits on comment-length:

internal/proxy/prompt_cache_key_retry_test.go L41–43: 3-line test function godoc restates what the function name (RetriesAndMemoizes) already encodes; the invariant fits in 2.
internal/translate/emit_openai_responses.go L81–83: stableResponsesPromptCacheKey godoc is 3 lines; the counterpart cross-reference adds no actionable WHY and the rest fits in 2.

workweave-bot

Advisory only — comment-length nits. Won't block merge.

claude · 2026-08-28T20:05:18Z

Claude finished @devin-ai-integration[bot]'s task —— View job

Run git diff to analyze added comments
Analyze comment blocks for verbosity
Post review or exit silently

No comment blocks of 3+ lines warranting a flag. All added blocks that reached 3+ lines were already reviewed and trimmed in this PR's previous commits. Remaining blocks explain genuinely non-obvious behavior concisely: each 3-line block carries distinct WHY content that cannot be cut further without losing the invariant.

View job

translate: send prompt_cache_key affinity hint to openai_gateway endp…

3047eb1

…oints

devin-ai-integration Bot assigned steventohme Aug 28, 2026

devin-ai-integration Bot requested a review from steventohme August 28, 2026 18:10

translate: send metadata.user_id affinity hint to anthropic_gateway e…

ca05bb5

…ndpoints

devin-ai-integration Bot changed the title ~~translate: send prompt_cache_key affinity hint to openai_gateway endpoints~~ translate: send session-affinity hints to openai_gateway and anthropic_gateway endpoints Aug 28, 2026

workweave-bot reviewed Aug 28, 2026

View reviewed changes

Comment thread internal/translate/emit_anthropic.go Outdated

translate: tighten applyAnthropicSessionAffinity doc comment

b58ff34

cursor Bot reviewed Aug 28, 2026

View reviewed changes

Comment thread internal/translate/emit_openai.go

devin-ai-integration Bot mentioned this pull request Aug 28, 2026

proxy: allow sibling failover for BYOK gateway turns via held gateway keys #1104

Merged

translate/proxy: strip prompt_cache_key after a gateway unknown-field…

055d7c5

… 400

cursor Bot reviewed Aug 28, 2026

View reviewed changes

Comment thread internal/proxy/service.go

steventohme added 2 commits August 28, 2026 19:20

proxy: extend prompt_cache_key strip-and-retry to the OpenAI Chat Com…

dff1166

…pletions ingress

Merge origin/main into devin/1787940603-gateway-session-affinity

8c4e6a3

Resolve internal/proxy/service.go: keep main's surface-based dispatchOpenAI refactor and re-apply the prompt_cache_key strip-and-retry (per-attempt StripPromptCacheKey, pre-commit retry, endpoint memoization) on top.

workweave-bot reviewed Aug 28, 2026

View reviewed changes

Comment thread internal/proxy/prompt_cache_key_retry_test.go Outdated

proxy: tighten test doc comment per review

7f26025

workweave-bot reviewed Aug 28, 2026

View reviewed changes

Comment thread internal/proxy/prompt_cache_key_retry_test.go Outdated

Comment thread internal/translate/envelope.go Outdated

Comment thread internal/translate/session_affinity_test.go Outdated

docs: tighten comments per review nits

a4fdc29

workweave-bot reviewed Aug 28, 2026

View reviewed changes

Comment thread internal/providers/provider.go Outdated

Comment thread internal/providers/provider_retry_test.go Outdated

Comment thread internal/proxy/service.go Outdated

docs: tighten comments per review nits

913bd66

workweave-bot reviewed Aug 28, 2026

View reviewed changes

Comment thread internal/proxy/service.go Outdated

workweave-bot reviewed Aug 28, 2026

View reviewed changes

Comment thread internal/proxy/service.go Outdated

Comment thread internal/translate/session_affinity_coverage_test.go Outdated

docs: tighten comments per review nits

a79ab21

cursor Bot reviewed Aug 28, 2026

View reviewed changes

Comment thread internal/proxy/service.go

translate: carry prompt_cache_key on the Responses emit path

d2112c8

workweave-bot reviewed Aug 28, 2026

View reviewed changes

Comment thread internal/translate/emit_openai_responses.go Outdated

Comment thread internal/translate/session_affinity_test.go Outdated

translate: tighten Responses affinity comments

2a0ac28

workweave-bot reviewed Aug 28, 2026

View reviewed changes

Comment thread internal/proxy/prompt_cache_key_retry_test.go Outdated

Comment thread internal/translate/emit_openai_responses.go Outdated

translate,proxy: trim comments per review

739b78b

steventohme enabled auto-merge (squash) August 28, 2026 20:05

steventohme merged commit 782e4b9 into main Aug 28, 2026
14 of 15 checks passed

steventohme deleted the devin/1787940603-gateway-session-affinity branch August 28, 2026 20:53

devin-ai-integration Bot mentioned this pull request Aug 28, 2026

smoke: exclude the per-run prompt_cache_key hint from cassette keys #1117

Merged

aminsamir45 mentioned this pull request Aug 28, 2026

fix(smoke): publish cassettes 0644 so the nightly refresh can commit them #1118

Closed

devin-ai-integration Bot mentioned this pull request Aug 28, 2026

translate: send x-grok-conv-id affinity header for grok via openai_gateway #1121

Merged

Uh oh!

Conversation

devin-ai-integration Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

greptile-apps Bot commented Aug 28, 2026

Uh oh!

devin-ai-integration Bot commented Aug 28, 2026

🤖 Devin AI Engineer

Uh oh!

claude Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

claude Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

workweave-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

claude Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

devin-ai-integration Bot commented Aug 28, 2026

Uh oh!

Uh oh!

claude Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

workweave-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

claude Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

workweave-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

claude Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

workweave-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

claude Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

workweave-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

workweave-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

claude Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

claude Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devin-ai-integration Bot commented Aug 28, 2026

Uh oh!

workweave-bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot commented Aug 28, 2026 •

edited

Loading

claude Bot commented Aug 28, 2026 •

edited

Loading

claude Bot commented Aug 28, 2026 •

edited

Loading

claude Bot commented Aug 28, 2026 •

edited

Loading

claude Bot commented Aug 28, 2026 •

edited

Loading

claude Bot commented Aug 28, 2026 •

edited

Loading

claude Bot commented Aug 28, 2026 •

edited

Loading

claude Bot commented Aug 28, 2026 •

edited

Loading

claude Bot commented Aug 28, 2026 •

edited

Loading

claude Bot commented Aug 28, 2026 •

edited

Loading

claude Bot commented Aug 28, 2026 •

edited

Loading

claude Bot commented Aug 28, 2026 •

edited

Loading