feat: add prompt caching support for OpenRouter provider#76
feat: add prompt caching support for OpenRouter provider#76danny-avila wants to merge 5 commits intodevfrom
Conversation
OpenRouter uses the same cache_control format as Anthropic's native API, so reuse addCacheControl() for cache breakpoint insertion. Also map cache_write_tokens to cache_creation in usage metrics.
|
@codex review |
|
Codex Review: Didn't find any major issues. What shall we delve into next? ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@danny-avila Dou you think this can be merged anytime soon? Currently Gemini models are slow as hell on Openrouter and cost of Anthropic models without caching is huge (as our use case requires a lot of tool calls). Thanks! |
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 22177b4eef
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| promptCacheProvider === Providers.OPENROUTER && | ||
| dynamicInstructions !== '' | ||
| ) { | ||
| body = [new HumanMessage(dynamicInstructions), ...body]; |
There was a problem hiding this comment.
When promptCache is enabled for OpenRouter, dynamicInstructions are moved into a prepended HumanMessage, but systemMessageTokens is still computed only from systemMessage. That makes instructionTokens underestimate prompt overhead, so pruning can admit too much history and the final request can exceed maxContextTokens in runs with large dynamic instructions (for example summary-in-system or long additional instructions). This is a regression specific to the new OpenRouter prompt-cache path because the dynamic segment is no longer included in the counted system message.
Useful? React with 👍 / 👎.
| promptCacheProvider === Providers.OPENROUTER && | ||
| dynamicInstructions !== '' | ||
| ) { | ||
| body = [new HumanMessage(dynamicInstructions), ...body]; |
There was a problem hiding this comment.
Prepending dynamicInstructions as the first non-system message makes the conversation opener change whenever those instructions vary (e.g., rotating summary text), which defeats OpenRouter’s sticky routing heuristic and drops prompt-cache hit rates for that session. The new behavior should keep the first non-system message stable and place volatile instructions later; otherwise prompt caching can silently degrade to repeated cache misses in dynamic-instruction workflows.
Useful? React with 👍 / 👎.
2 participants
OpenRouter uses the same cache_control format as Anthropic's native API, so reuse addCacheControl() for cache breakpoint insertion. Also map cache_write_tokens to cache_creation in usage metrics.