Skip to content

ci: bump Go to 1.26.6, and make the eval harness fail loudly when the model is never invoked - #342

Merged
qu0b merged 3 commits into
masterfrom
qu0b/ci-go-bump-and-eval-fail-loud
Aug 19, 2026
Merged

ci: bump Go to 1.26.6, and make the eval harness fail loudly when the model is never invoked#342
qu0b merged 3 commits into
masterfrom
qu0b/ci-go-bump-and-eval-fail-loud

Conversation

@qu0b

@qu0b qu0b commented Aug 19, 2026

Copy link
Copy Markdown
Member

Follow-up to #341, which merged with two red checks. Neither was caused by that
PR — both were already failing on master — so this fixes them separately.

1. govulncheck — fixed

scripts/govulncheck.sh has an empty ALLOWLIST and fails on any reachable
advisory, so it goes red whenever new stdlib advisories are published, with no
code change required. Seven landed against go1.26.5 and have blocked every PR
since master last went green on 2026-08-06:

Advisory Package Advisory Package
GO-2026-5026 net/http GO-2026-6089 net/http
GO-2026-5972 encoding/asn1 GO-2026-6090 crypto/tls
GO-2026-6088 encoding/xml GO-2026-6091 html/template
GO-2026-6218 net/url

All seven are standard-library and all are fixed in go1.26.6. Every workflow
resolves its toolchain from go-version-file: go.mod, so the go directive is
the single place to bump.

Dockerfile also pinned golang:1.26-bookworm by digest at go1.26.4, so
release binaries would keep shipping the vulnerable stdlib even after the go.mod
bump — repinned to the go1.26.6 digest (verified by running go version in
both images). Dockerfile.mcp and Dockerfile.proxy float on golang:1.26-*
and pick it up on their own.

Verified locally: scripts/govulncheck.sh now reports "no reachable
vulnerabilities"
, and make build + make test are green on 1.26.6.

2. Eval Smoke — diagnosis fixed, outage is external

Every smoke case comes back with 0 tokens, 0 tool calls and an empty answer,
and the harness reports crashed=False / "0 errors" — so an infrastructure
failure is graded as eight wrong answers. That is wrong twice over: it reads as
an eval regression when nothing about the eval changed, and crashed=False
skips promptfoo/provider.py's retry-on-crash path, which exists precisely for
transient provider faults.

Root cause in the harness: execute() only ever read text and tool parts
off the returned messages. A provider/model failure — which opencode reports as
an error on the message itself — was dropped on the floor, leaving an empty
ExecutionResult indistinguishable from a legitimately empty answer.

Now the harness captures that error, and treats a turn with no text, no tool
call and no tokens as an execution error. Both set is_error, so the run is
marked crashed, retried once, and reported with the provider's own message.

This will not turn the check green

The outage is external. Evidence:

  • The subject pin (opencode-go/deepseek-v4-flash) has not changed since
    2026-07-15 — git log -S on config/settings.py and eval-smoke.yaml is
    empty for that window, while the check went green 2026-07-21 → red 2026-08-06.
  • The panda server in the failing run is healthy (schemas fetched, indices
    built, zero errors in server.log); the artifact traces show
    crashed=False tokens=0 tools=0 with an empty answer.
  • The grader (qwen3.7-plus) rides the same gateway and the same
    OPENCODE_GO_API_KEY and worked fine — it spent 6,767 tokens grading those
    empty answers.

So deepseek-v4-flash appears to have gone away provider-side. What this PR
changes is that the next run will say so — printing the provider's error
instead of silently scoring zeros — which is what's needed to pick the
replacement pin.

I deliberately did not swap the model myself: the grader working proves the
gateway serves qwen3.7-plus for single-shot completion, not that it works as a
tool-calling agent subject, and I have no key to verify that. Guessing a pin
and calling CI fixed would be worse than leaving it legible. Happy to make the
swap once someone with the key confirms a working agent model.

Tests

tests/eval/tests/test_agent_no_output.py — 12 cases: provider-error surfacing,
the silent-empty-turn guard, the three "has some signal" shapes that must still
be accepted (answer only / tools only / tokens only), the provider error-payload
shapes, and three execute() end-to-end cases through a fake opencode client
asserting is_error and a preserved real answer.

Full eval suite green (86 passed). ruff check clean on the new file; the two
pre-existing UP038 findings in opencode_agent.py are untouched, and the file
was already unformatted on master so I did not reformat it into this diff.

qu0b added 2 commits August 19, 2026 13:55
scripts/govulncheck.sh has an empty ALLOWLIST and fails on any reachable
advisory, so it goes red whenever new stdlib advisories are published — no code
change required. Seven landed against go1.26.5 and have blocked every PR since
master last went green on 2026-08-06:

  GO-2026-5026  net/http        GO-2026-6089  net/http
  GO-2026-5972  encoding/asn1   GO-2026-6090  crypto/tls
  GO-2026-6088  encoding/xml    GO-2026-6091  html/template
  GO-2026-6218  net/url

All seven are standard-library and all are fixed in go1.26.6. Every workflow
resolves its toolchain from `go-version-file: go.mod`, so the go directive is
the single place to bump.

Dockerfile pinned golang:1.26-bookworm by digest at go1.26.4, so release
binaries would keep the vulnerable stdlib even after the go.mod bump; repin to
the go1.26.6 digest. Dockerfile.mcp and Dockerfile.proxy float on golang:1.26-*
and pick it up on their own.

Verified: `scripts/govulncheck.sh` reports "no reachable vulnerabilities", and
`make build` + `make test` are green on 1.26.6.
Every smoke case in CI comes back with 0 tokens, 0 tool calls and an empty
answer, and the harness reports `crashed=False` / "0 errors" — so an
infrastructure failure is graded as eight wrong answers. That is wrong twice
over: it reads as an eval regression when nothing about the eval changed, and
`crashed=False` skips promptfoo/provider.py's retry-on-crash path, which exists
precisely for transient provider faults.

execute() only ever read `text` and `tool` parts off the returned messages, so
a provider/model failure — which opencode reports as an `error` on the message
itself — was dropped on the floor and left an empty ExecutionResult that looked
like a legitimately empty answer.

Now: capture that error, and treat a turn with no text, no tool call and no
tokens as an execution error. Both set is_error, so the run is marked crashed,
retried once, and reported with the provider's own message.

This does not fix the outage behind the current red build. The subject model
pin (opencode-go/deepseek-v4-flash) has not changed since 2026-07-15, and the
grader rides the same gateway and key successfully, so the model appears to
have gone away provider-side between the 2026-07-21 green run and the
2026-08-06 red one. This change is what makes the next run say so.
@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

🐼 Smoke eval — 6050fa9: ❌ 0/8 pass

📊 Interactive report — tokens p50 0 · tokens/solve 0.

Reference points: v0.38.7 98% · master@30aac0d 0% · qu0b/ci-go-bump-and-eval-fail-loud@ca876ca 0%.

question result tokens tools
forky_node_coverage 💥 crash 0 0
tracoor_node_coverage 💥 crash 0 0
mainnet_block_arrival_p50 💥 crash 0 0
list_datasources 💥 crash 0 0
block_count_24h 💥 crash 0 0
missed_slots_24h 💥 crash 0 0
chartkit_default_arrival_distribution 💥 crash 0 0
storage_upload_session_scoped 💥 crash 0 0
🔭 Langfuse traces (8 runs; ⚠️ = failed)

The report walks this branch's commits against the master baseline and the most recent release. A self-contained copy is in the run's eval-smoke-* artifact.

The first CI run with the new guard surfaced the real payload:

  {"data": {"message": "The latest version of this model is only available
   hosted in China and requires explicit opt in: ...", "statusCode": ...}}

_error_text only looked for message/detail/error at the top level, so this fell
through to the raw-JSON fallback and got truncated at 300 chars. Look one level
down under "data" as well, so the reason reaches the CI table intact.
@qu0b

qu0b commented Aug 19, 2026

Copy link
Copy Markdown
Member Author

The new guard found the root cause on its first run

govulncheck is green on this branch. Eval Smoke is still red — but it now
says why, which it could not do before:

provider error for opencode-go/deepseek-v4-flash: {"data": {"message":
"The latest version of this model is only available hosted in China and
requires explicit opt in: https://opencode.ai/workspace/wrk_01KPYC.../go",
"provider_id": null, "statusCode": ...}}

Every case now reports CRASHED: provider error … instead of ✗ 0.00 with an
empty answer, and crashed=True in the saved traces (was crashed=False).

So the two-week outage is an opt-in flag, not a dead model. deepseek-v4-flash
moved its latest version to China-hosted and now requires an explicit opt-in on
the workspace. That matches the timeline exactly — nothing in this repo changed
since 2026-07-15, and the check went green 2026-07-21 → red 2026-08-06.

Two ways to fix it, both needing someone with workspace access:

  1. Opt in at the linked workspace settings and keep the current pin — the
    eval keeps measuring the same subject, so historical scores stay comparable.
  2. Repin DEFAULT_AGENT_MODEL in tests/eval/config/settings.py to a model
    that doesn't require the opt-in. Note this breaks score comparability with
    past runs.

I'd suggest (1) if the China-hosted routing is acceptable for CI, since it keeps
the measurement continuous.

I've pushed one more commit: the real payload nests the text under data, so
_error_text was falling through to the raw-JSON fallback and truncating at 300
chars. It now reads that shape directly, with a test using the exact payload
above.

@qu0b
qu0b merged commit 8955831 into master Aug 19, 2026
10 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant