Add structured output with tools support for models with limited structured output

[devv-shayan]

Add structured output with tools support for models with limited structured output

Summary

Adds an opt-in structured output with tools feature to enable using tools and structured outputs simultaneously on models that don't natively support both (specifically Google Gemini via LiteLLM).

Problem: Models like Gemini return BadRequestError: Function calling with a response mime type 'application/json' is unsupported when trying to use tools with structured outputs (via response_schema).

Solution: New enable_structured_output_with_tools parameter on the Agent class that:

• Injects JSON formatting instructions into the system prompt
• Disables native response_format to avoid API errors
• Parses the model's JSON response into the specified Pydantic model
• Defaults to False for backward compatibility
• Only actively used by LitellmModel (OpenAI models ignore it as they have native support)

Test plan

Unit tests (13 tests added):

• tests/utils/test_prompts.py - Tests for prompt generation and injection logic
  • Validates JSON schema extraction from Pydantic models
  • Tests conditional injection based on tools/output_schema presence
  • Verifies opt-in behavior (only injects when explicitly enabled)

Integration test:

• tests/test_gemini_local.py - Manual test script for Gemini (requires API key)
  • Tests actual Gemini integration with tools + structured output
  • Verifies structured output with tools works end-to-end

Verification performed:

make lint     # ✅ All checks passed
make format   # ✅ 359 files formatted correctly
make mypy     # ✅ No type errors
make tests    # ✅ 995 passed, 3 skipped, 20 failed*

*20 failures are pre-existing Windows-specific SQLite file locking issues in temporary directory cleanup (not related to this PR). All 13 new tests pass.

Test coverage:

• Updated all existing test mock models to accept enable_structured_output_with_tools parameter
• Verified backward compatibility (parameter is optional with default False)
• Tested with and without the feature enabled
• Validated structured output parsing with nested Pydantic models

Issue number

#2032

Checks

• I've added new tests (if relevant)
• I've added/updated the relevant documentation
• I've run make lint and make format
• I've made sure tests pass

Documentation

Added comprehensive documentation:

• docs/models/structured_output_with_tools.md - Complete guide with examples
  • Problem explanation with error messages
  • Step-by-step solution with code examples
  • "When to Use" table for different model providers
  • Debugging tips and best practices
• docs/agents.md - Added section on using structured outputs with tools
• docs/models/litellm.md - Added example for Gemini integration
• mkdocs.yml - Added new doc page to navigation

Files Changed

Source code:

• src/agents/agent.py - Added enable_structured_output_with_tools parameter
• src/agents/util/_prompts.py - New utility for JSON prompt generation
• src/agents/models/interface.py - Added parameter to Model interface
• src/agents/models/openai_chatcompletions.py - Pass-through (ignores parameter)
• src/agents/models/openai_responses.py - Pass-through (ignores parameter)
• src/agents/extensions/models/litellm_model.py - Active implementation
• src/agents/run.py - Passes parameter to model calls

Tests:

• tests/utils/test_prompts.py - 13 new unit tests
• tests/test_gemini_local.py - Integration test
• Updated 5 test mock models to match new interface

Documentation:

• docs/models/structured_output_with_tools.md - New comprehensive guide
• docs/agents.md - Added usage section
• docs/models/litellm.md - Added Gemini example
• mkdocs.yml - Updated navigation

Implementation Details

Design decisions:

1. Opt-in by default - enable_structured_output_with_tools=False ensures no impact on existing code
2. LiteLLM-focused - Only LitellmModel actively uses this; OpenAI models ignore it
3. Interface consistency - All models accept the parameter for uniform signatures
4. User control - Users explicitly enable based on their model's requirements

How it works:

1. User sets enable_structured_output_with_tools=True on Agent
2. LitellmModel checks if both tools and output_schema are present
3. If yes, generates JSON instructions from Pydantic model schema
4. Injects instructions into system prompt
5. Disables response_format to avoid API errors
6. Model returns JSON in text response
7. SDK parses and validates against Pydantic model

Example Usage

from agents import Agent, function_tool
from agents.extensions.models.litellm_model import LitellmModel
from pydantic import BaseModel, Field

class WeatherReport(BaseModel):
    city: str = Field(description="City name")
    temperature: float = Field(description="Temperature in Celsius")

@function_tool
def get_weather(city: str) -> dict:
    return {"city": city, "temperature": 22.5}

# This now works with Gemini!
agent = Agent(
    name="WeatherBot",
    model=LitellmModel("gemini/gemini-1.5-flash"),
    tools=[get_weather],
    output_type=WeatherReport,
    enable_structured_output_with_tools=True,  # Required for Gemini
)

Backward Compatibility

✅ Fully backward compatible - All changes are additive:

• New parameter has default value (False)
• Existing code works without modification
• OpenAI models ignore the parameter (use native support)
• Only affects users who explicitly enable it

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[seratch]

@devv-shayan Thanks for looking into this. However, as I mentioned at #2032 (comment), we prefer resolving this only by the changes on the LiteLLM or our LiteLLM adapter side. So, we don't accept this large diff for this.