Anthropic Prompt Caching: Fix SYSTEM_ONLY cache strategy to not expli… #4537
+464
−35
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@adase11 I made some slight semantic changes here. Could you take a look and see what you think? |
Can do, sorry I am traveling until next week but I'll try to get to it in a timely manner. I also saw your other question and am giving it some thought. Thanks! |
…citly cache tools This commit fixes a bug where the SYSTEM_ONLY cache strategy was incorrectly placing cache_control on tool definitions, making it functionally identical to SYSTEM_AND_TOOLS. Problem: - SYSTEM_ONLY and SYSTEM_AND_TOOLS both placed cache breakpoints on tools AND system messages, using 2 breakpoints - This violated the documented behavior and wasted breakpoints - The strategies were functionally identical Root Cause: - CacheEligibilityResolver.resolveToolCacheControl() checked if eligible message types included SYSTEM, which both strategies have - It should have checked the strategy directly instead Solution: - Changed resolveToolCacheControl() to use allowlist approach - Only SYSTEM_AND_TOOLS and CONVERSATION_HISTORY explicitly cache tools - SYSTEM_ONLY now places cache_control only on system message - Tools still get cached implicitly via Anthropic's cache hierarchy (tools → system → messages) Impact: - SYSTEM_ONLY: Uses 1 breakpoint (system only) instead of 2 - SYSTEM_AND_TOOLS: Uses 2 breakpoints (last tool + system) as before - Users can now optimize breakpoint usage more effectively Changes: - Fix CacheEligibilityResolver.resolveToolCacheControl() to check strategy directly instead of eligible message types - Enhance testSystemOnlyCacheStrategy() to include tools and verify they don't have cache_control - Update toolCacheControlRespectsStrategy() test to verify all strategies behave correctly Signed-off-by: Soby Chacko <[email protected]>
sobychacko
force-pushed
the
fix/anthropic-system-only-cache-strategy
branch
from
82d63b2 to
56e2e0a
Compare
This commit introduces the TOOLS_ONLY caching strategy, enabling scenarios where large tool definitions should be cached while system prompts remain dynamic and uncached. Key Changes: - Add TOOLS_ONLY enum to AnthropicCacheStrategy with comprehensive javadocs - Update CacheEligibilityResolver to support TOOLS_ONLY strategy - Enhance all strategy javadocs with detailed use cases and token guidance - Add comprehensive unit tests covering all caching scenarios - Update documentation with TOOLS_ONLY examples and cascade invalidation Use Cases: - Multi-tenant SaaS applications with shared tools but per-tenant system prompts - A/B testing scenarios with stable tools but variable system instructions - Applications with large tool sets (5000+ tokens) and dynamic contexts Technical Details: - TOOLS_ONLY uses 1 cache breakpoint on the last tool definition - System messages are NOT cached, processed fresh on each request - Supports Anthropic's cache hierarchy (tools → system → messages) - Compatible with cascade invalidation behavior Documentation: - Added strategy comparison table with breakpoint usage - Added multi-tenant SaaS use case example - Updated best practices with cascade invalidation explanation - Clarified SYSTEM_AND_TOOLS independence behavior Signed-off-by: Soby Chacko <[email protected]>
|
@sobychacko apologies for the delay. I'll do my best to get to this this week. |
|
@sobychacko - yep this looks good to me, thanks! |
|
Thanks, @adase11 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…citly cache tools
This commit fixes a bug where the SYSTEM_ONLY cache strategy was incorrectly placing cache_control on tool definitions, making it functionally identical to SYSTEM_AND_TOOLS.
Problem:
Root Cause:
Solution:
Impact:
Changes:
Thank you for taking time to contribute this pull request!
You might have already read the contributor guide, but as a reminder, please make sure to:
git commit -s) per the DCOmainbranch and squash your commitsFor more details, please check the contributor guide.
Thank you upfront!