Track agents@v0 release cut for eval measurement manifests

[ascerra]

Summary

Eval measurements (fullsend eval-measure) resolve stock-agent manifests from fullsend-ai/agents@v0. Activation is two-step:

1. Merge fullsend-ai/agents#722 (or equivalent) so eval/measurements/<agent>.yaml exists on agents main.
2. Cut a v0.x.y release that re-points the floating v0 tag.

Merging #722 alone does not activate managed GHA/GitLab measure steps — v0 stays on the prior release SHA until a cut.

Why it matters

Until step 2 lands, managed eval-measure jobs clean-skip (no remote manifest). Local FULLSEND_DIR overrides work today. Without a tracked release cut, measurements can stay silently provisional after the companion PRs merge.

Related

• fullsend ADR 0087 / guide: two-step activation wording
• Companion engine PR: feat(eval)!: add eval measurements and EM-001 trace_fitness scorer #6036
• Companion manifests PR: feat(eval): add measurement manifests for online scoring agents#722

Acceptance

• agents#722 (or successor) merged with stock eval/measurements/*.yaml
• v0.x.y release cut; v0 resolves to a SHA that contains those manifests
• Smoke: a managed job (or equivalent) produces eval-measurements.jsonl without a local FULLSEND_DIR override

[ascerra]

Linked from ADR 0087 / eval-measurements guide on #6036. Companion manifests: fullsend-ai/agents#722.

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 4:21 PM UTC · Completed 4:24 PM UTC

Commit: 18afad2 · View workflow run →

[fullsend-ai-triage]

This issue is blocked on upstream work that must land first:

Primary blocker: fullsend-ai/agents#722 — the measurement manifests PR — is still open and unmerged. Until it merges, there are no eval/measurements/*.yaml files on agents main, and the subsequent v0.x.y release cut cannot include them.

Additionally, the companion engine PR #6036 in this repo (which introduces fullsend eval-measure and the manifest resolution logic) is also still open. Both PRs need to land for the end-to-end flow to work, though the acceptance criteria here specifically gate on the agents-side activation.

Once agents#722 merges, the remaining steps (cutting a v0.x.y release and smoke-testing a managed job) are manual operational tasks that cannot be automated by a code agent.

Blocked by:

• feat(eval): add measurement manifests for online scoring agents#722

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 4:18 PM UTC · Completed 4:22 PM UTC

Commit: 2c7812a · View workflow run →

[fullsend-ai-triage]

Update: Both code prerequisites from the prior triage have been resolved — agents#722 and PR #6036 are now merged.

However, the latest agents release is v0.36.0 (published Aug 11), which predates agents#722's merge (Aug 20). The floating v0 tag still points to a SHA without the measurement manifests, so managed eval-measure jobs continue to clean-skip.

The remaining acceptance criteria (release cut + smoke test) are operational tasks in the agents repo. A tracking issue will be created there for the release cut. Once the release lands and v0 resolves to a SHA containing eval/measurements/*.yaml, the smoke test can proceed.

No code changes are needed in this repository — the engine-side work landed via PR #6036.

Blocked by:

• Cut v0.37.0 release to activate eval measurement manifests agents#892