eval: run the smoke subject through a LiteLLM proxy - #344
Merged
Conversation
Both ends of the eval are pinned to the opencode-go zen gateway, so when zen drops a model out from under us there is no second road. That is not hypothetical: deepseek-v4-flash went opt-in and every smoke case has come back empty since, which is why CI is red on master rather than on anything panda changed. A LiteLLM proxy is OpenAI-compatible, so both ends can use one. The judge takes a litellm/<model> spec, the same way it already takes codex/<model>. The subject needs more: opencode has no built-in entry for a private proxy, so declare it as an openai-compatible provider with the model under test, or opencode cannot resolve the id. Nothing changes for existing runs: a bare model name still grades through zen, codex/ still grades through the Responses API, and an opencode-go subject declares no provider block. Verified against a live proxy: the judge's exact request shape returns a well-formed rubric JSON through the public host.
|
Skipped automated review — PR is too large for a useful single-shot review.
Reviewed @ |
The zen pin is what is red, not panda: deepseek-v4-flash went opt-in provider-side, so every smoke case comes back with no tokens and no answer. Point the subject at starflinger-anthropic through the proxy, with a key scoped to that one model and capped at $25/30d. The judge stays on zen — its path never broke, and a Qwen judge over a Claude subject keeps the two families apart, which is the property the judge choice is documented to need. starflinger-openai would have been the other candidate; the proxy currently answers it with "No fallback model group found", so it is not an option today.
|
Skipped automated review — PR is too large for a useful single-shot review.
Reviewed @ |
Contributor
🐼 Smoke eval —
|
| question | result | tokens | tools |
|---|---|---|---|
forky_node_coverage |
✅ | 109,137 | 9 |
tracoor_node_coverage |
✅ | 147,523 | 11 |
mainnet_block_arrival_p50 |
✅ | 67,252 | 6 |
list_datasources |
✅ | 50,170 | 3 |
block_count_24h |
✅ | 150,009 | 12 |
missed_slots_24h |
✅ | 106,466 | 11 |
chartkit_default_arrival_distribution |
✅ | 887,879 | 19 |
storage_upload_session_scoped |
✅ | 287,265 | 18 |
evm_faucet_fund_throwaway_address |
✅ | 96,807 | 9 |
🔭 Langfuse traces (9 runs; ⚠️ = failed)
The report walks this branch's commits against the master baseline and the most recent release. A self-contained copy is in the run's eval-smoke-* artifact.
qu0b
changed the title
The new smoke case caught the failure it was written for: given "I need test ETH for a throwaway address", the agent spent 18 tool calls, never found evm.faucet, and invented an HTTP faucet API (POST /api/claim) with a placeholder address. The function was documented but nothing an agent scans first said the module funds anything: the module description listed execution, tracing, submission and assembly, and no example was named after funding. Say it in the description, and add the two-line example that is the whole answer.
|
Skipped automated review — PR is too large for a useful single-shot review.
Reviewed @ |
Six agent sessions in flight meant six concurrent conversations through the gateway, each one a long tool-using run — that is a lot of simultaneous load to put on a shared proxy for a job whose point is a sanity check, and it makes a rate limit or a budget cap look like an eval regression. Run them sequentially instead. The smoke job's timeout goes to 60 minutes to pay for it: nine runs took 5m32s at six in flight, and the slowest single run is the floor when there is only one.
|
Skipped automated review — PR is too large for a useful single-shot review.
Reviewed @ |
eval-smoke moved to the proxy; the full-eval and release-qualification jobs did not, so the next dispatch or release would have failed exactly the way smoke did. release-eval pinned deepseek-v4-flash explicitly, eval inherited it through the harness default. Fix it at the layer that owns it — DEFAULT_AGENT_MODEL and DEFAULT_SUBJECTS now name proxy models, so local runs and the harden loop stop reaching for a model that answers with nothing — and give both jobs the proxy credentials. The harden pair stays two families (Claude and MiniMax) so a harness change still cannot pass by suiting one vendor, and the judge deliberately stays on opencode-go: a separate gateway from the subjects, and the half of the eval that never broke. The CI key's scope now covers both subject models. The usage examples in the README, the module docstring and the settings help all named the dead model too.
|
Skipped automated review — PR is too large for a useful single-shot review.
Reviewed @ |
Drop minimax as a subject, in the defaults, the tests and the README example, and narrow the CI key back to the one model. That leaves the harden loop with a single subject, which is a real loss: the pair existed so a harness change had to help two vendors rather than suit one. Worth restoring when a second non-Claude model is available — starflinger-openai would be it, once the proxy stops answering it with "No fallback model group found". Also stop failing the smoke job on rubric misses. It exists to prove the pipeline runs end to end, and it now goes red for the reasons that mean the harness is broken — no tokens, provider errors, crashes — rather than when a graded agent flubs one of nine questions at k=1. Two runs today both scored 78%, failing different cases each time, with clean transports underneath; release-eval already draws this line with --min-pass 0.
|
Skipped automated review — PR is too large for a useful single-shot review.
Reviewed @ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Both ends of the eval were pinned to the opencode-go zen gateway, so when zen drops a model there is no second road. That is what has been happening:
deepseek-v4-flashwent opt-in provider-side, every smoke case returns 0 tokens and a provider error, andEval Smokehas been red on master since 2026-08-06 — nothing to do with panda.What
A LiteLLM proxy is OpenAI-compatible, so both ends can use one.
--judge-model litellm/<model>grades throughLITELLM_PROXY_URLwithLITELLM_PROXY_API_KEY, the same waycodex/<model>already selects the Responses API.--subject litellm/<model>:cli. opencode has no built-in entry for a private proxy, so the generated config declares one (@ai-sdk/openai-compatibleplus the model under test); without that models map opencode cannot resolve the id.litellm/starflinger-anthropic:cli. The judge stays on zen: its path never broke, and a Qwen judge over a Claude subject keeps the families apart, which is the property the judge choice is documented to need.starflinger-openaiwas the other candidate. The proxy currently answers it withNo fallback model group found for original model_group=starflinger-openai, so it is not an option today.Credentials
LITELLM_PROXY_API_KEY(repo secret) andLITELLM_PROXY_URL(repo variable) are set. The key is scoped tostarflinger-anthropicand capped at $25 / 30d.Verified
94 passed){"pass": true, "score": 1, "reason": "ok"}starflinger-anthropicanswers through the scoped keycodex/still through the Responses API, anopencode-gosubject renders noproviderblockNot verified locally: a full subject run through the proxy —
opencodeis not installed on this machine, so the provider block is covered by rendering tests. This PR's ownEval Smokerun is the end-to-end check, and it is the point of the change.