Skip to content

fix(openai): cache stable prompts explicitly - #620

Draft
solvemproblr wants to merge 1 commit into
mainfrom
asa/openai-explicit-prompt-caching
Draft

fix(openai): cache stable prompts explicitly#620
solvemproblr wants to merge 1 commit into
mainfrom
asa/openai-explicit-prompt-caching

Conversation

@solvemproblr

@solvemproblr solvemproblr commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Issue

GPT-5.6 implicit prompt caching writes through changing user input, so independent rollouts cannot reuse a shared system prompt and may pay for cache writes that never hit.

Solution

  • Put the stable system prompt in a developer content block with an explicit cache breakpoint for GPT-5.6 and later.
  • Use explicit-only caching so changing rollout input remains outside the cached prefix.
  • Expose prompt_cache_key for high-volume request grouping and retain cache_write_tokens in normalized usage.
  • Preserve the existing request shape for earlier GPT models.

Validation

  • uv run --extra modal pytest -q — 1116 passed
  • uv run ruff check .
  • uv run ruff format . --check
  • uv run ty check hud/agents/openai/agent.py hud/agents/types.py hud/agents/tests/test_openai_agent.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant