ci: bump Go to 1.26.6, and make the eval harness fail loudly when the model is never invoked - #342
Conversation
scripts/govulncheck.sh has an empty ALLOWLIST and fails on any reachable advisory, so it goes red whenever new stdlib advisories are published — no code change required. Seven landed against go1.26.5 and have blocked every PR since master last went green on 2026-08-06: GO-2026-5026 net/http GO-2026-6089 net/http GO-2026-5972 encoding/asn1 GO-2026-6090 crypto/tls GO-2026-6088 encoding/xml GO-2026-6091 html/template GO-2026-6218 net/url All seven are standard-library and all are fixed in go1.26.6. Every workflow resolves its toolchain from `go-version-file: go.mod`, so the go directive is the single place to bump. Dockerfile pinned golang:1.26-bookworm by digest at go1.26.4, so release binaries would keep the vulnerable stdlib even after the go.mod bump; repin to the go1.26.6 digest. Dockerfile.mcp and Dockerfile.proxy float on golang:1.26-* and pick it up on their own. Verified: `scripts/govulncheck.sh` reports "no reachable vulnerabilities", and `make build` + `make test` are green on 1.26.6.
Every smoke case in CI comes back with 0 tokens, 0 tool calls and an empty answer, and the harness reports `crashed=False` / "0 errors" — so an infrastructure failure is graded as eight wrong answers. That is wrong twice over: it reads as an eval regression when nothing about the eval changed, and `crashed=False` skips promptfoo/provider.py's retry-on-crash path, which exists precisely for transient provider faults. execute() only ever read `text` and `tool` parts off the returned messages, so a provider/model failure — which opencode reports as an `error` on the message itself — was dropped on the floor and left an empty ExecutionResult that looked like a legitimately empty answer. Now: capture that error, and treat a turn with no text, no tool call and no tokens as an execution error. Both set is_error, so the run is marked crashed, retried once, and reported with the provider's own message. This does not fix the outage behind the current red build. The subject model pin (opencode-go/deepseek-v4-flash) has not changed since 2026-07-15, and the grader rides the same gateway and key successfully, so the model appears to have gone away provider-side between the 2026-07-21 green run and the 2026-08-06 red one. This change is what makes the next run say so.
🐼 Smoke eval —
|
| question | result | tokens | tools |
|---|---|---|---|
forky_node_coverage |
💥 crash | 0 | 0 |
tracoor_node_coverage |
💥 crash | 0 | 0 |
mainnet_block_arrival_p50 |
💥 crash | 0 | 0 |
list_datasources |
💥 crash | 0 | 0 |
block_count_24h |
💥 crash | 0 | 0 |
missed_slots_24h |
💥 crash | 0 | 0 |
chartkit_default_arrival_distribution |
💥 crash | 0 | 0 |
storage_upload_session_scoped |
💥 crash | 0 | 0 |
🔭 Langfuse traces (8 runs; ⚠️ = failed)
The report walks this branch's commits against the master baseline and the most recent release. A self-contained copy is in the run's eval-smoke-* artifact.
The first CI run with the new guard surfaced the real payload:
{"data": {"message": "The latest version of this model is only available
hosted in China and requires explicit opt in: ...", "statusCode": ...}}
_error_text only looked for message/detail/error at the top level, so this fell
through to the raw-JSON fallback and got truncated at 300 chars. Look one level
down under "data" as well, so the reason reaches the CI table intact.
The new guard found the root cause on its first run
Every case now reports So the two-week outage is an opt-in flag, not a dead model. Two ways to fix it, both needing someone with workspace access:
I'd suggest (1) if the China-hosted routing is acceptable for CI, since it keeps I've pushed one more commit: the real payload nests the text under |
Follow-up to #341, which merged with two red checks. Neither was caused by that
PR — both were already failing on
master— so this fixes them separately.1.
govulncheck— fixedscripts/govulncheck.shhas an emptyALLOWLISTand fails on any reachableadvisory, so it goes red whenever new stdlib advisories are published, with no
code change required. Seven landed against
go1.26.5and have blocked every PRsince
masterlast went green on 2026-08-06:net/httpnet/httpencoding/asn1crypto/tlsencoding/xmlhtml/templatenet/urlAll seven are standard-library and all are fixed in
go1.26.6. Every workflowresolves its toolchain from
go-version-file: go.mod, so thegodirective isthe single place to bump.
Dockerfilealso pinnedgolang:1.26-bookwormby digest at go1.26.4, sorelease binaries would keep shipping the vulnerable stdlib even after the go.mod
bump — repinned to the go1.26.6 digest (verified by running
go versioninboth images).
Dockerfile.mcpandDockerfile.proxyfloat ongolang:1.26-*and pick it up on their own.
Verified locally:
scripts/govulncheck.shnow reports "no reachablevulnerabilities", and
make build+make testare green on 1.26.6.2.
Eval Smoke— diagnosis fixed, outage is externalEvery smoke case comes back with 0 tokens, 0 tool calls and an empty answer,
and the harness reports
crashed=False/ "0 errors" — so an infrastructurefailure is graded as eight wrong answers. That is wrong twice over: it reads as
an eval regression when nothing about the eval changed, and
crashed=Falseskips
promptfoo/provider.py's retry-on-crash path, which exists precisely fortransient provider faults.
Root cause in the harness:
execute()only ever readtextandtoolpartsoff the returned messages. A provider/model failure — which opencode reports as
an
erroron the message itself — was dropped on the floor, leaving an emptyExecutionResultindistinguishable from a legitimately empty answer.Now the harness captures that error, and treats a turn with no text, no tool
call and no tokens as an execution error. Both set
is_error, so the run ismarked crashed, retried once, and reported with the provider's own message.
This will not turn the check green
The outage is external. Evidence:
opencode-go/deepseek-v4-flash) has not changed since2026-07-15 —
git log -Sonconfig/settings.pyandeval-smoke.yamlisempty for that window, while the check went green 2026-07-21 → red 2026-08-06.
built, zero errors in
server.log); the artifact traces showcrashed=False tokens=0 tools=0with an empty answer.qwen3.7-plus) rides the same gateway and the sameOPENCODE_GO_API_KEYand worked fine — it spent 6,767 tokens grading thoseempty answers.
So
deepseek-v4-flashappears to have gone away provider-side. What this PRchanges is that the next run will say so — printing the provider's error
instead of silently scoring zeros — which is what's needed to pick the
replacement pin.
I deliberately did not swap the model myself: the grader working proves the
gateway serves
qwen3.7-plusfor single-shot completion, not that it works as atool-calling agent subject, and I have no key to verify that. Guessing a pin
and calling CI fixed would be worse than leaving it legible. Happy to make the
swap once someone with the key confirms a working agent model.
Tests
tests/eval/tests/test_agent_no_output.py— 12 cases: provider-error surfacing,the silent-empty-turn guard, the three "has some signal" shapes that must still
be accepted (answer only / tools only / tokens only), the provider error-payload
shapes, and three
execute()end-to-end cases through a fake opencode clientasserting
is_errorand a preserved real answer.Full eval suite green (86 passed).
ruff checkclean on the new file; the twopre-existing
UP038findings inopencode_agent.pyare untouched, and the filewas already unformatted on master so I did not reformat it into this diff.