Skip to content

Conversation

TomeHirata
Copy link
Collaborator

@TomeHirata TomeHirata commented Sep 22, 2025

Closes #8834

Cache Image.format for better inference time throughput.

@TomeHirata TomeHirata requested a review from Copilot September 22, 2025 03:39
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR optimizes adapter methods by adding LRU cache decorators to improve inference time throughput. The changes cache the results of formatting methods that depend only on signature types rather than user inputs.

  • Adds @lru_cache(maxsize=64) decorator to formatting methods across adapter classes
  • Targets methods that format field structures, descriptions, and task information
  • Focuses on methods that are signature-dependent but input-independent for safe caching

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
dspy/adapters/json_adapter.py Adds caching to field structure and output requirements formatting methods
dspy/adapters/chat_adapter.py Adds caching to field description, structure, and task description formatting methods
dspy/adapters/base.py Adds caching to the demo formatting method in the base adapter class

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@TomeHirata TomeHirata closed this Sep 22, 2025
@TomeHirata TomeHirata changed the title Cache adapter methods for better throuput Cache adapter methods for better throughput Sep 22, 2025
@TomeHirata TomeHirata reopened this Sep 22, 2025
@TomeHirata TomeHirata changed the title Cache adapter methods for better throughput Cache Image.format for better throughput Sep 22, 2025
@TomeHirata TomeHirata merged commit 50185ca into stanfordnlp:main Sep 22, 2025
15 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Performance: caching formatted signatures
2 participants