✨ feature: live backend smoke for contributor CLIs + durable run telemetry

[Danathar]

Summary

Hive runs contributor tasks by driving vendor coding-agent CLIs ("clankers" — Claude Code, Codex, …) through the contributor relay, and that integration seam breaks whenever a vendor ships a CLI update: every existing test pins the seam against captured fixtures, so vendor drift lands green in CI and fails contributors in production. This PR adds a live smoke harness that runs the real relay against a fake hub — and, on a 6-hour schedule with real credentials, the real CLIs on a one-line task — plus durable per-run telemetry so the same health questions are answerable for real fleet traffic, not just synthetic runs.

• The harness caught three live relay bugs on its first run, all fixed here: interactive claude completions never matched the verdict parser (wrong bullet glyph — every one silently fell back to chrome-idle detection), and headless codex hung until timeout (stdin never closed) and refused non-git workspaces (missing --skip-git-repo-check).
• The scheduled lane files deduplicated per-lane issues: a latest-lane red means vendor drift is incoming; a pinned-lane red means shipped contributor images are broken today.
• Blast radius: relay completion/headless behavior (the three fixes above), one new hub JSONL log + one new public read-only API endpoint, new CI workflow + a keyless merge-gating subset; no change to routing, cooldowns, trust, or any hub decision path.

Implements the test-harness proposal from maintainer discussion (Discord, 2026-08-31); no tracking issue exists.

Detail

bin/test_backend_smoke.sh (also just backend-smoke) — three tiers:

• Keyless drift checks: the relay's HEADLESS_BACKENDS table vs KNOWN_BACKENDS in config/backends.conf (previously sync-by-comment only), and codex/claude version-pin parity across src/Dockerfile and src/Dockerfile.contributor (the contributor codex pin was an inline hardcode with no guard; now an ARG).
• Keyless stub wire-contract scenarios: a stub CLI binary on PATH drives the full relay↔hub loop (handshake, task_assign, verdict parse off one-shot output, task_failed shape) with zero API spend. This subset runs merge-gating in v2-ci.
• Live per-backend scenarios: detect_cli health probe, headless end-to-end (claude -p / codex exec, asserting the HIVE_VERDICT: no_work_needed reply on the wire), and interactive end-to-end (real CLI in tmux, relay scraping the pane, asserting completion_signal == "verdict" — chrome_idle is a failure naming the sentinel-contract regression class). Credentials: API key, base64 subscription-login blob (HIVE_SMOKE_*_B64, the 🐛 bug: contribute-k8s generates a workload with no agent-CLI credential — all five allow-listed headless backends launch unauthenticated #5103 decode shape), or the operator's own login copied into a throwaway HOME. Skips are loud-but-green locally and fatal under HIVE_TEST_REQUIRE_BACKEND_SMOKE=1 (the test: run the entrypoint behavioural suites in the container lane, and fail on a skip (#5380) #5383 inversion).

.github/workflows/backend-smoke.yml — cron 45 2,8,14,20 * * * (off the taken :00/:30 slots), matrix {claude, codex} × {latest, pinned}, per-arm credential scoping, cheapest model tiers (haiku / gpt-5.4-mini ≈ 32 one-line calls/day), create-issue-on-failure copied from v2-tests' schedule-gated dedupe shape but per lane. First and only workflow holding model credentials — deliberately its own file, outside the arm64 lane's no-secrets contract. Gated to github.repository == 'kubestellar/hive'.

Relay fixes (each found live by the harness, each with a regression test):

• detectHiveVerdict accepted only codex's • (U+2022) bullet; Claude Code renders ● (U+25CF). The pane showed the sentinel while the relay logged "no HIVE_VERDICT yet". Both glyphs now match; same anchoring and echo guards.
• runHeadlessTask never closed the child's stdin, and codex exec blocks on stdin-EOF even with the prompt in argv → zero-output hang until the 4h timeout. Stdin is now closed for every one-shot child.
• codex headless argv gains --skip-git-repo-check: exec refuses a non-git cwd, and the task workspace root is one until the agent's first clone.

Telemetry (src/pkg/dashboard/task_run_log.go): one JSONL record per accepted task_complete/task_failed in /data/contributors/task_runs.jsonl (0600, 10 MiB cap, single rotation) — the already-normalized fields (completion_signal, failure kind, verdict) plus wall-clock duration (new taskAssignedAt anchor, separate from lastLeaseRenew which progress reports move) and a derived scenario from a closed vocabulary (verdict_complete, idle_complete, headless_complete, env_failure, task_failure, unspecified_failure). GET /api/contribute/run-stats aggregates per backend (scenario counts, chrome-idle share, duration p50/p95); public read-only like the sibling contribute GETs, aggregates only. DECLARE-only per the contribute_protocol.go boundary — nothing routes on any of it; writes are best-effort and never fail a task. Documented in dashboard/openapi.json (route-parity test enforced).

Deferred follow-ups: fleet-level aggregation of per-hive run-stats over the heartbeat (touches hub.RepoActivityWire), backend=/model= extraction in activity_collector.go, and populating InvocationMeta.Session on contributor PRs (the correlation key the records already carry as session).

Post-merge to activate the scheduled lane: provision per-vendor secrets (either ANTHROPIC_API_KEY / OPENAI_API_KEY, or HIVE_SMOKE_CLAUDE_CREDENTIALS_B64 / HIVE_SMOKE_CODEX_AUTH_B64) and run one workflow_dispatch per lane; the pinned in-container lane could not be exercised locally.

Related issues

None — implements the maintainer-proposed clanker test harness (Discord discussion, 2026-08-31). Touches the surfaces of #5376 (completion signal), #5335 (CODEX_HOME), #5383 (skip inversion), #5103 (delivered credentials).

Testing

• cd src && go build ./...
• cd src && go test -short -race ./pkg/dashboard/ (green, including new scenario-matrix, rotation, aggregation, and end-to-end WebSocket run-record tests)
• Other / not run (explain): bash bin/test_backend_smoke.sh — 33/33 with live claude (haiku) and codex (gpt-5.4-mini) arms, both headless and interactive; also green via the B64 subscription-credential path with an empty HOME. node bin/contributor-relay.test.js 256/256. bin/test_bin_suites_wired.sh, src/scripts/check-release-lines.sh, shellcheck on the new suite: all pass. The pinned in-container workflow lane needs a post-merge workflow_dispatch (no image credentials locally).

Contributor checklist

• PR targets v2 unless a maintainer requested another branch. (Targets v4, the current default branch.)
• Title uses the repo emoji convention, for example 📖 docs: ..., 🐛 fix: ..., or ✨ feature: ....
• Commits include DCO sign-off (git commit -s).
• Docs, examples, and policies are updated when behavior changes.
• CHANGELOG.md has an entry for user-visible changes (features, fixes, new env vars, behavior changes), or the change is not user-facing.
• No secrets, credentials, or local runtime state are committed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Jo76R9XXJ2iq6YcmVUiX2u

[kubestellar-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign clubanderson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kubestellar-hive]

Quality verification (live hive host): fetched pull/5408/head and ran:

• go test ./pkg/dashboard/ -run 'TaskRun|RunStats|RunLog|Scenario' — 5/5 pass (scenario table, append/stamp defaults, 10 MiB rotation, aggregation, end-to-end WS run-record).
• node bin/contributor-relay.test.js — 256/256 pass, including the new verdict-glyph, stdin-close, and --skip-git-repo-check regression tests.
• bash bin/test_backend_smoke.sh — keyless drift tier 13/13 pass; wire-contract tier skipped loudly (no npm network access here), live tiers skipped as designed.

Skip behavior is loud-but-green as documented and the HIVE_TEST_REQUIRE_BACKEND_SMOKE=1 inversion is present. No concerns from the quality side; live/pinned lanes still need the post-merge workflow_dispatch noted in the PR body.

Filed by quality agent (hold-gated mode).

🐝 Hive Agent: quality | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=quality backend=copilot model=claude-fable-5