Add built-in Foundry adapter - #6
Open
eedorenko wants to merge 17 commits into
Open
Conversation
Move the Foundry agent adapter from a standalone custom adapter file into beval as a built-in protocol (protocol: foundry). Add cortyx eCommerce analytics cases (100 cases across 11 files) for evaluating Foundry agents with deterministic graders. - Add python/src/beval/adapters/foundry.py with lazy azure SDK imports - Register "foundry" protocol in adapters/__init__.py - Add "foundry" optional dependency group in pyproject.toml - Update samples/foundry-agent to use protocol: foundry - Remove standalone beval_foundry_adapter.py - Add run.sh convenience script Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- run.sh accepts optional cases path argument (defaults to cases/cortyx) - Update README install URL to point to current branch Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cases live in the cortyx repo (microsoft/cortyx/cases/). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Revert uv.lock changes - Restore eval.config.yaml to standard sample config - Clean up README to be generic (no branch-specific URLs) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
eedorenko
changed the title
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a first-party protocol: foundry adapter to beval so Microsoft Foundry agents can be evaluated without the standalone custom-adapter module, and updates the Foundry sample to use the built-in protocol.
Changes:
- Added
FoundryAdapter(lazy-importingazure-ai-projects/azure-identity) and registered thefoundryprotocol in adapter dispatch. - Introduced a
foundryoptional dependency group inpyproject.tomland included it in theallextra. - Updated
samples/foundry-agent/to useprotocol: foundryand simplified the sample run instructions.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| samples/foundry-agent/README.md | Updates sample documentation to use the built-in foundry protocol and revised setup/run instructions. |
| samples/foundry-agent/eval.config.yaml | Switches agent definition to protocol: foundry and uses built-in connection keys. |
| samples/foundry-agent/agent.yaml | Switches agent definition to protocol: foundry and simplifies connection config. |
| python/src/beval/adapters/foundry.py | Adds the built-in Foundry adapter implementation using azure-ai-projects. |
| python/src/beval/adapters/init.py | Registers foundry as a known protocol and wires it into create_adapter(). |
| python/pyproject.toml | Adds foundry extras and updates all and mypy overrides accordingly. |
Comments suppressed due to low confidence (3)
python/src/beval/adapters/foundry.py:57
self._timeoutis read from the agent definition but never applied to the Foundry/OpenAI request. As a result,timeout:inagent.yaml/eval.config.yamlhas no effect and requests may hang indefinitely depending on underlying SDK defaults.
python/src/beval/adapters/foundry.py:28- New built-in protocol adapter
FoundryAdapteris introduced without unit tests (unlike ACP/A2A adapters). Adding tests for lazy-import failure messaging, requiredendpointvalidation, and the basic request path (mocking the SDK client) would help prevent regressions.
python/src/beval/adapters/foundry.py:68 - When
adapter_input.queryis a message list, this code flattens it into a single string (and overwritesquery). Other adapters extract the last user message instead, which avoids accidentally including assistant/system content and preserves the originalqueryvalue forSubject.input. Consider using a separatequery_textand extracting the last user turn.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Azure AI Foundry project endpoints use the OpenAI v1 API path (/openai/v1/chat/completions) rather than the Azure-native path that requires api-version query params. Without this, the judge fails with "Missing required query parameter: api-version" or "API version not supported". Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add effective_latency (geometric mean of time+1, capped at 100s), p95_latency, fail_rate, keyword_recall, and accuracy_rate with binomial standard errors to RunSummary. These metrics appear in results.json summary when computed, and are omitted when not applicable (backward compatible). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bug fix for redacted keyword_recall metrics
Regex fix for keyword_recall being redacted
Strip comma thousands separators (e.g. 1,000 → 1000) from both the keyword and the agent response before matching. This fixes false negatives where a case expects "9786" but the agent responds with "9,786.07". Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Strip trailing decimal zeros from both keyword and output before matching (e.g. 104.90 → 104.9, 1.00 → 1). Combined with the existing comma normalization, this handles all common number formatting variations without losing meaningful precision. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a numeric keyword isn't found literally, check if rounding the more-precise value to the less-precise one's scale produces a match. Works both ways: keyword 1.577 matches output 1.58, and keyword 9786 matches output 9786.07. Limited to at most 2 decimal places difference to avoid overly aggressive matching. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Foundry entra_id path was using plain OpenAI client with a static bearer token baked into headers. After 60 minutes the token expired, causing 401 errors for all remaining cases. Switch to AzureOpenAI client with azure_ad_token_provider so tokens refresh automatically on every request. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
protocol: foundryas a built-in adapter in beval for evaluating Microsoft Foundry agents, replacing the standalone custom adapter approach.Changes
python/src/beval/adapters/foundry.py— new adapter usingazure-ai-projectsSDK with lazy importspython/src/beval/adapters/__init__.py— register"foundry"in protocol dispatch and known protocolspython/pyproject.toml— addfoundryoptional dependency group (azure-ai-projects,azure-identity); include inallextrasamples/foundry-agent/— update sample to useprotocol: foundryinstead ofprotocol: custom; remove standalone adapter fileTest plan
uv run pytest tests/test_adapters.py -v— all 42 tests pass🤖 Generated with Claude Code