feat(eval): add eval measurements and EM-001 trace_fitness scorer - #6036
feat(eval): add eval measurements and EM-001 trace_fitness scorer#6036ascerra wants to merge 18 commits into
Conversation
|
🤖 Finished Review · ✅ Success · Started 11:45 AM UTC · Completed 12:02 PM UTC Commit: |
PR Summary by QodoAdd eval measurements CLI and EM-001 trace_fitness scoring
AI Description
Diagram
High-Level Assessment
Files changed (31)
|
Site previewPreview: https://6cfb80c2-site.fullsend-ai.workers.dev Commit: |
Code Review by Qodo
1.
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
ReviewFindingsMedium
Low
Info
Previous runReviewFindingsLow
Previous run (2)ReviewFindingsLow
Previous run (3)ReviewFindingsLow
Previous run (4)ReviewFindingsLow
Previous run (5)ReviewFindingsLow
Previous run (6)ReviewFindingsLow
Previous run (7)ReviewFindingsHigh
Low
Next steps:
Previous run (8)ReviewFindingsLow
Previous run (9)ReviewFindingsLow
Previous run (10)ReviewFindingsCritical
High
Medium
Low
Next steps:
Previous run (11)ReviewFindingsHigh
Medium
Low
Labels: PR adds new Go eval-measure package (internal/evalmeasure/), modifies CI workflows (.github/workflows/), and adds documentation (docs/guides/, docs/ADRs/) Next steps:
|
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
fullsend-ai-review
Bot
added
type/feature
|
🤖 Review · ❌ Terminated · Started 1:34 PM UTC · Ended 1:54 PM UTC Commit: |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
🤖 Finished Review · ✅ Success · Started 1:34 PM UTC · Completed 1:54 PM UTC Commit: |
- Fix data-loss bug: swap AppendMeasurements before RecordScored so ledger only marks scored after measurement is persisted - Fix broken curl header args in action.yml using bash array instead of parameter expansion (word-split safe) - Add Prerequisites section to eval-measurements guide - Add OTEL/OTLP inline definitions on first use in guide - Use > **Planned:** callout format for unimplemented OTLP export - Update docs/guides/README.md with eval measurements entry - Consolidate ADR 0087 Decision into single paragraph (was multiple sub-decisions); trim Consequences to 5 bullets - ADR 0050 cross-reference uses Planned callout format - Rename printResults → printMeasurementResults Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
🤖 Review · Commit: |
|
🤖 Finished Review · ✅ Success · Started 1:15 AM UTC · Completed 1:33 AM UTC Commit: |
Superseded by updated review
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
🤖 Finished Review · ✅ Success · Started 1:45 AM UTC · Completed 2:02 AM UTC Commit: |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
waynesun09
left a comment
waynesun09
left a comment
There was a problem hiding this comment.
Review-only sweep (no approval, no change request) on the current head. Six findings posted inline: 2 HIGH on the GitLab scaffold job, 4 MEDIUM across find.go precedence, EM-001 skip classification, the guide's declarative example, and the ADR's activation dependency. Each was checked against the existing comment threads on this PR and is distinct from the issues already answered there.
| case "${EVENT_TYPE:-}" in | ||
| issue_*) | ||
| export GITLAB_ISSUE_URL="${CI_SERVER_URL}/${CI_PROJECT_PATH}/-/issues/${STATUS_IID:-0}" | ||
| if [[ -n "${STATUS_IID:-}" && "${STATUS_IID}" != "0" ]]; then |
There was a problem hiding this comment.
[HIGH] Conditional GITLAB_ISSUE_URL export leaves the var truly unset, which strict harness env validation rejects before the sandbox starts
In-diff (verified against origin/main...origin/feat/eval-measurements): the unconditional export GITLAB_ISSUE_URL=".../-/issues/${STATUS_IID:-0}" (and the merge_requests equivalent) is replaced by a conditional export, so on the issue path when STATUS_IID is empty/"0", and on the MR path when both CI_MERGE_REQUEST_IID and STATUS_IID are empty/"0", GITLAB_ISSUE_URL is left truly unset rather than set to a bogus value.
That variable is consumed under fullsend's strict env gate. On fullsend-ai/agents@main, harness/triage.yaml:79 declares forge.gitlab.env.runner.ISSUE_URL: ${GITLAB_ISSUE_URL} (and :83 the sandbox equivalent); harness/code.yaml:120,125 does the same. internal/harness/harness.go:652-703 ValidateRunnerEnvWith documents and enforces "Variables set to an empty string are allowed; only truly unset variables produce an error", returning "%s: host variable %s is not set (referenced in %q)", and internal/cli/run.go:575 calls it before any expansion and before the sandbox is created (StepFail + validating env: ...).
Timing nuance: the current agents v0 tag (6bdcab69, 2026-08-10) has no gitlab: forge block and no env/gitlab/triage.env at all, so this does not detonate against today's v0 pin — it lands the moment the GitLab harness support on agents@main reaches a cut v0.x.y release (the same release cut this PR's measurement manifests are waiting on), or immediately for any consumer pinned to agents@main or carrying a local harness copied from it. This is the same lenient-to-strict env delivery regression class that broke agents#211.
The paired scaffold test only asserts the string shape ("${STATUS_IID}" != "0"); nothing asserts the harness env still validates.
Suggestion: Keep the intent (no invented .../issues/0) but export an empty value on every path instead of leaving the variable unset — e.g. initialise GITLAB_ISSUE_URL="" before the case, assign inside each branch only when the IID is real/non-zero, and export GITLAB_ISSUE_URL unconditionally after the esac. An empty string satisfies ValidateRunnerEnvWith, and it does not reintroduce greenwashing: resolveWorkItemID() (internal/cli/run.go:2416) does strings.TrimSpace(os.Getenv("GITLAB_ISSUE_URL")) and skips empty, so the run still records fullsend.work_item_id = "unknown" and EM-001's work_item check still fails. Add a scaffold assertion that export GITLAB_ISSUE_URL is present on every path.
There was a problem hiding this comment.
Fixed in 8029af1.
Always initialize GITLAB_ISSUE_URL=\"\", set a real URL only when the IID is non-empty/non-zero, and export GITLAB_ISSUE_URL unconditionally after the esac. Empty still satisfies harness env validation and does not invent …/issues/0 for EM-001. Scaffold asserts GITLAB_ISSUE_URL=\"\" and export GITLAB_ISSUE_URL.
| --fullsend-dir .fullsend \ | ||
| --target-repo . \ | ||
| --output-dir /tmp/fullsend-output \ | ||
| --output-dir "${CI_PROJECT_DIR}/output" \ |
There was a problem hiding this comment.
[HIGH] GitLab run output directory is now nested inside --target-repo, so host telemetry is uploaded into the sandbox and is committable by the agent
The PR moves GitLab's output from /tmp/fullsend-output to ${CI_PROJECT_DIR}/output (needed for the new artifacts: paths: [output/] block), but the same job passes --target-repo ., which resolves to $CI_PROJECT_DIR.
Verified ordering in internal/cli/run.go: step 3 creates runDir = <outputBase>/agent-<name>-<pid>-<ts> (run.go:903-910) and installs the file span exporter on it; step 8 then copies the target repo into the sandbox with sandbox.UploadDir(sandboxName, hostRepositoryDir, remoteRepositoryDir) (run.go:1260), an unfiltered tarball of everything under the repo root, tracked or not. agentWorkingDirExcludes (run.go:105-111) contains only .agentready/ and .fullsend-workspace/, so output/ is excluded neither from the tarball nor from the sandbox .git/info/exclude written by excludeAgentWorkingDirs (run.go:1302).
Net effect on GitLab only (GitHub Actions keeps output/ and the checkout as siblings under GITHUB_WORKSPACE): every span already flushed by the SimpleSpanProcessor file exporter (sandbox_create, pre-script) plus the runDir itself is handed to the sandboxed agent, and an agent doing git add -A can commit run telemetry into a consumer MR.
The guide's only mitigation is prose telling operators to gitignore output/, but internal/scaffold/fullsend-repo-gitlab/ ships no .gitignore at all (the tree contains only .fullsend/config.yaml, .gitlab-ci.yml and .gitlab/ci/*).
Suggestion: Make output/ a sibling of the checkout rather than a child of --target-repo while staying under $CI_PROJECT_DIR (e.g. --target-repo "${CI_PROJECT_DIR}/target-repo" with output at ${CI_PROJECT_DIR}/output), or add output/ to agentWorkingDirExcludes so it is excluded from both the sandbox tarball and the sandbox git index — and ship the mitigation with the scaffold (a .gitignore containing output/, plus a scaffold test asserting it) instead of only documenting it in the guide.
There was a problem hiding this comment.
Fixed in 8029af1.
Added output/ to agentWorkingDirExcludes (sandbox .git/info/exclude) and exclude output/ from the UploadDir tarball so host telemetry is not uploaded or git-addable. Scaffold embeds a recommended .gitignore fragment with output/ (tested); Collect skips installing it as a root file so it cannot overwrite a consumer ignore list.
| // It never walks deeper, so an agent-planted | ||
| // iteration-N/output/run-telemetry.jsonl is ignored. | ||
| func FindPlatformTelemetry(outputDir, agent string) ([]string, error) { | ||
| direct := filepath.Join(outputDir, PlatformTelemetryFile) |
There was a problem hiding this comment.
[MEDIUM] FindPlatformTelemetry's root-level short-circuit outranks the real agent runDir, newly reachable now that output/ lives in the repo
FindPlatformTelemetry stats <outputDir>/run-telemetry.jsonl first and returns it immediately (find.go:33-37), before the os.ReadDir branch that applies the agent-<name>-<pid>-<unix> pattern and the --agent filter (find.go:39-71). So a file literally named run-telemetry.jsonl at the base of --output-dir wins over the genuine runDir even when --agent is passed.
This is distinct from the already-answered threads on this file: the nested iteration-N/output/ copy case (fixed in c8438d83) and the sibling/leftover runDir case (fixed in 5a218846) both concern the ReadDir branch. Root-level precedence has not been raised on this PR.
It becomes reachable because of the paired GitLab change in this same PR: --output-dir is now ${CI_PROJECT_DIR}/output inside the target-repo working tree, and nothing excludes or gitignores output/, so an agent that commits output/run-telemetry.jsonl gets it restored by the CI checkout of a later pipeline — where it sits at exactly the path this short-circuit trusts, silently displacing the real run's telemetry for scoring.
Suggestion: Prefer the runDir-scoped file over a root-level file of the same name: skip the direct short-circuit when --agent is set and a matching child runDir exists (or gate the short-circuit on outputDir's own basename matching hostRunDirPattern). Add a test with a planted root-level run-telemetry.jsonl alongside a real agent-<name>-<pid>-<unix> runDir to lock the precedence in.
There was a problem hiding this comment.
Fixed in 8029af1.
FindPlatformTelemetry prefers matching child runDirs over a root-level run-telemetry.jsonl. If a matching agent-<name>-… directory exists (even empty), the planted root file is ignored. Tests cover planted-root+runDir precedence and empty matching runDir.
| // No agent span: run never reached an iteration (sandbox/provider/image | ||
| // failure). Exclude from pass/(pass+fail) so EM-001 trends measure the | ||
| // telemetry contract, not runner health. | ||
| if len(agents) == 0 { |
There was a problem hiding this comment.
[MEDIUM] EM-001 scores a hard-killed run as fail rather than skip when agent spans flushed but the run span never ended
ScoreFitnessNamed excludes runner-health failures only through the len(agents) == 0 early return (fitness.go:53-74). The root run span's attributes (exit_code, gen_ai.request.model, fullsend.cost_usd, fullsend.num_turns, iterations) are set in a deferred closure at internal/cli/run.go:941-975 and the span is exported only when it ends. The file exporter is a SimpleSpanProcessor (internal/telemetry/telemetry.go:153), so agent spans hit run-telemetry.jsonl the instant they end — meaning a SIGKILL / OOM-kill / job timeout after at least one agent span produces a telemetry file with agent spans and no run span.
With hasRun false, run is the zero-value Span, so span_tree (fitness.go:84, requires hasRun), identity (identityOK reads run's fullsend.agent, :145-149), work_item (:161-164), operation (attrNonEmpty on run, :90) and exit all fail, while model/usage/cost_tools_turns can still pass off the agent spans — roughly 3/8, recorded as fail, not skip.
Both the action.yml step and the GitLab step are fail-open and run after the agent regardless of its exit status, so this hits exactly the jobs most likely to be killed, and the repo already treats hard-kill as a real scenario (the "Finalize orphaned status comment" step exists for it). The result is that runner health systematically depresses the EM-001 pass rate that ADR 0087 says measures the telemetry contract. This is the complement of the len(agents) == 0 case fixed in 5a218846, not the same case.
Suggestion: Treat "agent spans present but no root run span" the same way as len(agents) == 0: return LabelSkip with an explanation such as root run span missing; run terminated before flush — or require hasRun before scoring the 8-check contract at all.
There was a problem hiding this comment.
Fixed in 8029af1.
When agent spans are present but the root run span is missing (hard kill / timeout before flush), EM-001 returns label: skip with explanation root run span missing; run terminated before flush — same exclusion class as no agent span.
| checks: | ||
| - name: turn_token_ratio | ||
| assert: ratio_lte | ||
| numerator: gen_ai.usage.total_tokens |
There was a problem hiding this comment.
[MEDIUM] Guide's declarative-manifest example uses two attribute names that fullsend never emits
The "logic-as-config (future declarative engine)" YAML block documents the intended shape of a planned feature with numerator: gen_ai.usage.total_tokens and denominator: fullsend.turns (lines 129-130). Neither key exists:
gen_ai.usage.total_tokensis not an OTel GenAI semantic-convention attribute (the v1.37.0 GenAI registry definesinput_tokens/output_tokens) and fullsend never emits it —internal/cli/run.goemits onlygen_ai.usage.input_tokens,output_tokens,cache_creation.input_tokens,cache_read.input_tokensand conditionallyreasoning_tokens(run.go:961-971, 2346-2354).- The emitted turns attribute is
fullsend.num_turns(run.go:966), which is also whatdocs/guides/infrastructure/distributed-tracing.mddocuments and what this PR's own scorer reads viaAttrFullsendNumTurnsininternal/evalmeasure/fitness.go.
A not-yet-built config surface documented with invented keys is how wrong names get copied into the first real manifests written against agents#722.
Suggestion: Rewrite the example against attributes that are actually emitted — e.g. numerator: gen_ai.usage.output_tokens, denominator: fullsend.num_turns, or derive a total explicitly from input_tokens + output_tokens — and add a one-line note that the declarative block is illustrative and its key names are not yet a contract.
There was a problem hiding this comment.
Fixed in 8029af1.
Declarative example now uses emitted attrs gen_ai.usage.output_tokens and fullsend.num_turns, with an explicit note that the block is illustrative and not yet a contract.
| vendor-specific score adapters in core. `fullsend` owns the parser, scorers, | ||
| CLI, and GHA step; `fullsend-ai/agents` owns per-agent measurement manifests | ||
| (`eval/measurements/<agent>.yaml`) that declare which scorers to enable. | ||
| Stock-agent defaults resolve from `agents@v0` at runtime; local files are for |
There was a problem hiding this comment.
[MEDIUM] Activation is a two-step dependency (merge and a v0 release cut) that nothing tracks, and no run has exercised the resolved-manifest path
ADR 0087:88 states stock-agent defaults "resolve from agents@v0 at runtime" and the PR body frames the companion fullsend-ai/agents#722 merge as the only remaining dependency ("Until those land on agents@v0, the measure step skips cleanly").
Verified upstream on 2026-08-19: eval/measurements/ does not exist on fullsend-ai/agents at main or at v0 (GET contents/eval/measurements → 404; git ls-tree -r origin/main → no match), and #722 is still open. v0 is a floating major tag re-pointed only when a v0.x.y release is cut — it currently resolves to 6bdcab69 (2026-08-10) while main is at 816b89be (2026-08-19), nine days stale. So merging #722 alone activates nothing; a release must additionally be cut and v0 re-pointed.
Consequently there is no end-to-end evidence in this PR that scoring produces rows on the shipped code path: manifest resolution returns a clean skip today, the action.yml and GitLab steps are guaranteed no-ops, the PR's own Testing checklist leaves that box unchecked, and the only e2e artifact offered (MLflow screenshots) covers the MLflow adapter that was removed in 67aee649 and is no longer in this diff. Nothing in CI would surface "measurements have been silently no-op for N weeks". This is distinct from the existing GetRef/token/rate-limit threads.
Suggestion: State the two-step dependency explicitly in ADR 0087 and the guide ("manifests must be merged and included in a v0.x.y release that re-points v0") and open/link a tracking issue for the release cut. Land at least one check that exercises the resolved-manifest path before merge — e.g. an e2e/behaviour case pointing --fullsend-dir at a fixture containing eval/measurements/<agent>.yaml and asserting eval-measurements.jsonl appears in the uploaded artifact — or mark the GHA/GitLab wiring explicitly provisional until the manifests are actually on v0.
There was a problem hiding this comment.
Fixed in 8029af1.
ADR 0087 and the guide now state the two-step activation (merge manifests and cut a v0.x.y that re-points v0); managed wiring is marked provisional until then. Tracking: #6384. Added TestEvalMeasureCmd_LocalFullsendDirManifestProducesJSONL for the local FULLSEND_DIR resolved-manifest path.
|
Tracking the two-step |
Export empty GITLAB_ISSUE_URL on every GitLab path, keep host output/ out of the sandbox tarball and git exclude, prefer runDir telemetry over planted roots, skip EM-001 when the run span never flushed, and document two-step agents@v0 activation (#6384) plus planned Level 3 content scorers. Signed-off-by: Adam Scerra <ascerra@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
Pushed |
|
🤖 Review · Commit: |
Bring the branch current with main after the Aug-19 measurement review push. Signed-off-by: Adam Scerra <ascerra@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
🤖 Review · Commit: |
Pick up main commits landed after the previous sync.
|
🤖 Finished Review · ✅ Success · Started 7:07 PM UTC · Completed 7:24 PM UTC Commit: |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
waynesun09
left a comment
waynesun09
left a comment
There was a problem hiding this comment.
Review-only sweep on the current head (no approval, no change request) — 6 findings posted inline: 2 HIGH on the new output/ exclusion path (tar exclude anchoring differs between bsdtar and GNU tar; the exclusion is unconditional across forges and repos), and 4 MEDIUM on doc/implementation drift, the temp-dir fetch cache root, and the duplicated OTLP JSON schema. Each was checked against the existing threads on this PR; where a finding follows on from an already-answered thread, the inline comment says so.
| continue | ||
| } | ||
| // Exclude the directory itself and its contents at the archive root. | ||
| tarArgs = append(tarArgs, "--exclude=./"+pattern, "--exclude=./"+pattern+"/*") |
There was a problem hiding this comment.
[HIGH] UploadDir exclude patterns are not directory-anchored on bsdtar — every nested output/ dir is silently dropped from the sandbox copy on macOS
The new variadic exclude support builds --exclude=./<pattern> and --exclude=./<pattern>/*, and the code comment claims it excludes "the directory itself and its contents at the archive root." That is only true for GNU tar. exec.Command("tar", ...) resolves to bsdtar on any macOS host (local fullsend run, and macos-* runners), where the match is non-anchored.
Verified empirically against a fixture tree containing output/tracked.txt, src/a.go, sub/output/keep.txt, keep-root.txt, using the exact argv the code builds:
- bsdtar 3.5.3 / libarchive 3.7.4 (
/usr/bin/taron macOS): archive contains only./,./keep-root.txt,./src/,./src/a.go,./sub/—./sub/output/keep.txtis GONE. - GNU tar 1.35 (debian container, same fixture and flags): archive contains
./sub/output/,./sub/output/keep.txt— only the root./outputis dropped, as the comment claims.
So on macOS, fullsend run now silently omits any directory named output at any depth (frontend/output/, docs/output/, testdata/output/, …) from the repo copy handed to the agent. .git IS uploaded in the same tarball (confirmed: the exclude list is the only filter, and the COPYFILE_DISABLE comment immediately below exists precisely because AppleDouble files corrupt .git after a round-trip), so those tracked files appear as deletions in the sandbox's git status. .git/info/exclude does not mask deletions of tracked paths, so an agent running git add -A / git commit -a commits the removal.
The new regression test TestUploadDir_ExcludesPatternsFromTarball (internal/sandbox/sandbox_test.go:676) only asserts that a root-level output/ is absent; it has no nested-directory case, so this platform divergence passes CI on both platforms and is untested. In-diff: verified against merge-base 17df6eb — UploadDir gained the excludes ...string variadic and the --exclude loop in this PR.
Suggestion: Stop relying on tar's exclude-matching semantics, which differ across implementations. Enumerate the top-level entries of localPath with os.ReadDir, drop the excluded names, and pass the surviving entries as explicit ./<name> members instead of . — deterministic on bsdtar, GNU tar, and busybox alike. Add a regression case that creates sub/output/keep.txt and asserts it survives while root output/ is dropped. If --exclude is kept, the "at the archive root" comment must be corrected, since it is false for bsdtar (GNU's --anchored is not accepted by bsdtar).
There was a problem hiding this comment.
Fixed in 8596989.
Stopped using tar --exclude (bsdtar matches output at any depth). UploadDir now enumerates top-level members and omits excluded basenames only, so nested sub/output/ survives. Nested exclude paths are rejected. Regression test asserts nested keep + top-level drop.
| copyStart := time.Now() | ||
| printer.StepStart("Copying project code into sandbox") | ||
| if err := sandbox.UploadDir(sandboxName, hostRepositoryDir, remoteRepositoryDir); err != nil { | ||
| if err := sandbox.UploadDir(sandboxName, hostRepositoryDir, remoteRepositoryDir, "output/"); err != nil { |
There was a problem hiding this comment.
[HIGH] output/ is excluded from the sandbox tarball and .git/info/exclude unconditionally, for every forge and every repo
sandbox.UploadDir(sandboxName, hostRepositoryDir, remoteRepositoryDir, "output/") hardcodes the literal "output/" with no relationship to where the run output actually lives, and agentWorkingDirExcludes (run.go:113) adds the same literal to the in-sandbox .git/info/exclude for every run.
The motivating problem is GitLab-only, and I verified the asymmetry at head: the GitLab scaffold passes --target-repo . with --output-dir "${CI_PROJECT_DIR}/output" (fullsend-agent.yml:390-393), so the run dir nests inside the checkout. On GitHub, action.yml:407-408 keeps them disjoint — --output-dir "${GITHUB_WORKSPACE}/output" while --target-repo defaults to ${GITHUB_WORKSPACE}/target-repo (action.yml:388) — so the exclusion there is pure collateral, as it is for any local run with --output-dir outside the repo.
For a consumer repo that versions a top-level output/ directory (build artifacts, generated docs, ML runs, test fixtures — a generic enough name that this is realistic), the tarball omits the tracked files while .git is still uploaded, so the sandbox working tree shows them as deleted. .git/info/exclude does not mask deletions of tracked paths, so git status reports D output/... and an agent doing git add -A / git commit -a commits the removal. Nothing in the diff scopes this to the forge or layout that needs it.
Note the existing answered thread at internal/scaffold/fullsend-repo-gitlab/.gitlab/ci/fullsend-agent.yml:393 introduced this fix; this finding is a critique of the fix's implementation and has no comment on internal/cli/run.go. In-diff: both the "output/" entry in agentWorkingDirExcludes and the UploadDir argument are added by this PR (verified against merge-base 17df6eb).
Suggestion: Derive the exclusion from the actual run layout instead of a hardcoded name: compute rel, err := filepath.Rel(hostRepositoryDir, outputBase) and only pass an exclude (and only append to .git/info/exclude) when err == nil && filepath.IsLocal(rel) — i.e. when the output base genuinely sits inside the target repo — using rel as the pattern. The existing entries (.agentready/, .fullsend-workspace/) are fullsend-reserved names; output/ is not, so it needs the path check. This fix alone does not resolve the bsdtar anchoring bug above; both are needed.
There was a problem hiding this comment.
Fixed in 8596989.
Removed hardcoded output/ from agentWorkingDirExcludes. outputDirExcludeRel only excludes when --output-dir is a single-segment child of --target-repo (GitLab nested layout); sibling layouts (GHA) are unchanged. That relative name is passed to UploadDir and .git/info/exclude.
|
|
||
| - What signals matter most — cost, latency, token usage, action logs, decision traces, or something else? | ||
| - ~~How do we balance detailed tracing (useful for debugging) with the volume of data agents will produce?~~ Decided in [ADR 0050](ADRs/0050-distributed-tracing-instrumentation.md): instrument all lifecycle steps comprehensively; volume is managed by backends not by suppressing data at the source. | ||
| - ~~How do we score wild agent traces for trends without a second export stack?~~ Decided in [ADR 0087](ADRs/0087-eval-measurements-online-trace-scoring.md): eval measurements write local JSONL beside telemetry; portable remote export uses the same OTLP config as traces (planned); local eval-measurements.jsonl is always written. |
There was a problem hiding this comment.
[MEDIUM] Living doc contradicts eval-measurements write behavior: "local eval-measurements.jsonl is always written"
The 'Open questions' section records the resolved answer as "...local eval-measurements.jsonl is always written." This contradicts ADR 0087, the eval-measurements guide, and the implementation.
Verified in internal/evalmeasure/export_local.go: AppendMeasurements opens the file with O_APPEND|O_CREATE only after an early if len(results) == 0 { return nil } — so with zero new rows the file is never created. Upstream of that, runEvalMeasure skips entirely when telemetry or the manifest is missing (resolveEvalMeasureRegistry returns "" on a 404, resolveEvalMeasureTelemetry returns no paths), and a fully-ledgered rerun produces no new results. So the file is absent, not empty, in every one of those cases — the opposite of what the current-truth doc tells operators.
Not covered by the existing docs/architecture.md:290 thread, which was about the > **Planned:** callout format for OTLP export and is already marked Resolved.
Suggestion: Reword line 314 to say the local JSONL is written only when new measurement rows are produced (including label: skip rows), matching ADR 0087 and the user guide, so operators reading the current-truth doc are not misled into expecting the file unconditionally.
There was a problem hiding this comment.
Fixed in 8596989.
docs/architecture.md now matches ADR/guide/implementation: local eval-measurements.jsonl is written only when new measurement rows are produced (including label: skip); absent when telemetry/manifest missing, no traces match, or everything is already ledgered.
| - name: turn_token_ratio | ||
| assert: ratio_lte | ||
| numerator: gen_ai.usage.output_tokens | ||
| denominator: fullsend.num_turns |
There was a problem hiding this comment.
[MEDIUM] Incomplete fix in 8029af1: the declarative example still names an attribute fullsend never emits on the span it scopes to
Follow-up on the answered thread at this file:129 ("declarative attr names", replied "Fixed in 8029af1"). The fix replaced two nonexistent names, but introduced a different instance of the same class of error, and the guide now makes a stronger claim than the example supports: "attribute names below match what fullsend emits today" (line 121-123).
The block scopes itself with where: span: agent (line 136-137) and then uses numerator: gen_ai.usage.output_tokens (line 139) and denominator: fullsend.num_turns (line 140). gen_ai.usage.output_tokens is indeed on agent spans, but fullsend.num_turns is not. Verified by grepping every emit site: it is set exactly once, at internal/cli/run.go:969, inside the runCount > 0 block of the root-span telemetry defer. agentSpanEndAttrs (internal/cli/run.go:2343-2360) emits iteration, exit_code, gen_ai.system, gen_ai.request.model, input/output/cache tokens, fullsend.cost_usd and fullsend.tool_calls — no num_turns.
This PR's own fixtures confirm the split: in internal/evalmeasure/testdata/complete.jsonl the run span carries fullsend.num_turns while the agent span carries only fullsend.cost_usd / fullsend.tool_calls. So the illustrative manifest, as written, would evaluate a missing attribute on the very span it selects.
Suggestion: Either change the denominator to an attribute that exists on agent spans (fullsend.tool_calls or iteration), or change where: to select the root run span so fullsend.num_turns is in scope. If the example is meant to stay purely illustrative, drop or soften the "match what fullsend emits today" claim so it is not read as a per-span contract.
There was a problem hiding this comment.
Fixed in 8596989.
Declarative example now uses where: span: run so gen_ai.usage.output_tokens and fullsend.num_turns are both on the scoped span. Softened the surrounding note to say attrs match the run span today and are not a contract.
| func evalMeasureFetchContext(fullsendDir string, offline bool, printer *ui.Printer) (harness.ComposeOpts, forge.Client) { | ||
| workspace := fullsendDir | ||
| if workspace == "" { | ||
| workspace = os.TempDir() |
There was a problem hiding this comment.
[MEDIUM] eval-measure without --fullsend-dir uses os.TempDir() as the fetch cache root and audit-log location
evalMeasureFetchContext falls back to workspace = os.TempDir() when fullsendDir is empty (evalmeasure.go:198-201), and that value becomes both ComposeOpts.WorkspaceRoot and the base of AuditLogPath (filepath.Join(abs, ".fullsend-cache", "fetch-audit.jsonl")). fetch.CachePath builds <workspaceRoot>/.fullsend-cache/resources/sha256/<hash> (internal/fetch/cache.go:44-49, 162).
Verified reachable: --fullsend-dir defaults to "" (evalmeasure.go:79), and resolveEvalMeasureRegistry calls evalMeasureFetchContext(opts.fullsendDir, ...) on the remote-manifest path whenever no local override resolved (evalmeasure.go:172). So a plain fullsend eval-measure --agent X --telemetry <path> writes /tmp/.fullsend-cache/... and /tmp/.fullsend-cache/fetch-audit.jsonl.
/tmp is world-writable with the sticky bit, and this is a fixed, predictable path shared by every user on the host. On a shared runner or dev box the first invoking user creates the tree with their ownership and subsequent users' writes fail — and because the path is fail-open, the failure is silent. A local attacker can also pre-create /tmp/.fullsend-cache, or plant fetch-audit.jsonl as a symlink, to deny service or redirect the audit log. fullsend run never does this; it always has a real fullsend directory, and both managed workflows pass --fullsend-dir, which is why this is MEDIUM rather than HIGH — it bites direct CLI use only.
Not covered by the existing evalmeasure.go:204 / :208 threads, which I read in full: those concern the token warn-string wording and the --offline flag, not the workspace root.
Suggestion: Use a per-user location (os.UserCacheDir() plus a fullsend subdirectory) or a per-invocation os.MkdirTemp("", "fullsend-evalmeasure-*") removed on exit, instead of the shared os.TempDir() root. Alternatively require --fullsend-dir whenever the remote manifest fetch would be attempted, and skip cleanly otherwise.
There was a problem hiding this comment.
Fixed in 8596989.
Empty --fullsend-dir now uses os.UserCacheDir()/fullsend/eval-measure (0700) instead of shared os.TempDir(), with a pid-scoped temp fallback if cache dir is unavailable. Test pins the UserCacheDir path under a temp HOME.
| "os" | ||
| "strconv" | ||
| ) | ||
|
|
There was a problem hiding this comment.
[MEDIUM] OTLP JSON structs are an unpinned second copy of the writer's types, with no test binding reader to writer
internal/evalmeasure/parse.go:11-40 redeclares otlpTracesData / otlpResourceSpans / otlpScopeSpans / otlpSpan / otlpStatus / otlpKeyValue as a private copy of the identically-named types in internal/telemetry/fileexporter.go:71-124. Nothing links them — verified they are two independent declarations in two packages, and that no test in internal/evalmeasure imports internal/telemetry or drives the real exporter (grep over internal/evalmeasure/*_test.go finds only hand-built run-telemetry.jsonl temp files).
The assumptions do hold against the current writer — hex traceId/spanId rather than base64, startTimeUnixNano/endTimeUnixNano as strings, intValue as a string, doubleValue as a number, status.code as an integer enum. The gap is that none of it is asserted anywhere: internal/evalmeasure/testdata/README.md states the fixtures are synthetic and for unit tests only, and every fixture is hand-authored.
If the exporter ever switches to protojson (which encodes enums as strings like "STATUS_CODE_ERROR" and bytes as base64), json.Unmarshal fails on the whole line, every line lands in stats.SkippedLines, and eval-measure fail-opens to "no traces" with no test failure — exactly the silent degradation EM-001 exists to catch. This PR already uses the guard-the-invariant-with-a-test pattern for the far smaller filename-constant coupling, so the schema coupling being unguarded is the outlier. Distinct from the answered parse.go:84/:85/:87 threads, which are all about fail-open handling of corrupt lines and spans.
Suggestion: Add a round-trip test in internal/evalmeasure that drives the real exporter (or a telemetry-package golden generated from it) through ParseTelemetryFile, asserting stats.SkippedLines == 0 plus the EM-001 attributes. Better still, export the OTLP JSON types from internal/telemetry (or a small shared package) so there is one definition instead of two that can drift. Span.StatusCode is parsed but read by no scorer — either drop it or cover it in that round-trip.
There was a problem hiding this comment.
Fixed in 8596989.
Added TestParseTelemetryFile_RoundTripFromExporter: real telemetry.Setup file exporter → ParseTelemetryFile → EM-001 pass, with SkippedLines/SkippedSpans asserted zero. Shared OTLP types remain a follow-up; the round-trip gates encoding drift.
Enumerate UploadDir members instead of tar --exclude so nested output/ survives on bsdtar, scope host output exclusion to in-repo layouts, fix architecture/declarative docs, use UserCacheDir for eval-measure fetch cache, and round-trip the file exporter through ParseTelemetryFile. Signed-off-by: Adam Scerra <ascerra@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the branch current after the evening measurement review fix.
|
🤖 Review · ❌ Terminated · Started 1:20 AM UTC · Ended 1:41 AM UTC Commit: |
|
Pushed |
| } | ||
|
|
||
| direct := filepath.Join(outputDir, PlatformTelemetryFile) | ||
| if st, err := os.Stat(direct); err == nil && !st.IsDir() { |
There was a problem hiding this comment.
[low] edge-case
hostRunDirPattern regex uses greedy .+ for agent name capture. For a hypothetical all-digit agent name, the directory name would parse incorrectly due to greedy backtracking.
| ScorerFitness = "trace_fitness" | ||
| LabelPass = "pass" | ||
| LabelFail = "fail" | ||
| LabelSkip = "skip" |
There was a problem hiding this comment.
[low] cross-package-coupling
UnknownSentinel is defined in the scoring package but its meaning originates in the CLI layer (resolveWorkItemID). The dependency direction is inverted.
| ) | ||
|
|
||
| const ( | ||
| MeasurementsFile = "eval-measurements.jsonl" |
There was a problem hiding this comment.
[low] missing-doc-comment
Exported constants MeasurementsFile and LedgerFile lack doc comments, while the sibling PlatformTelemetryFile in find.go has one.
| // MeasureFile parses telemetry, scores with the manifest, and writes local | ||
| // eval-measurements.jsonl. Idempotent per ledger. | ||
| func MeasureFile(telemetryPath, registryPath, outDir string) ([]EvaluationResult, error) { | ||
| r, _, err := MeasureAndExport(context.Background(), telemetryPath, registryPath, outDir) |
There was a problem hiding this comment.
[low] api-shape
MeasureFile wraps MeasureAndExport but silently discards ParseStats. Only used in tests.
| ) | ||
|
|
||
| // AppendMeasurements writes one NDJSON EvaluationResult per line. | ||
| func AppendMeasurements(path string, results []EvaluationResult) (retErr error) { |
There was a problem hiding this comment.
[low] exported-api-surface
AppendMeasurements, AlreadyScored, and RecordScored are exported but only called within the evalmeasure package.
| if err != nil { | ||
| return nil, false, err | ||
| } | ||
| if registry == "" { |
There was a problem hiding this comment.
[low] error-handling
evalMeasureFetchContext workspace fallback: if UserCacheDir() and MkdirAll both fail, subsequent fetch operations will silently skip. Correct fail-open behavior.
| ledgerPath := filepath.Join(outDir, LedgerFile) | ||
| measPath := filepath.Join(outDir, MeasurementsFile) | ||
| var all []EvaluationResult | ||
| hook, _ := ctx.Value(persistHookKey{}).(func()) |
There was a problem hiding this comment.
[low] race-condition
Idempotency flow (AlreadyScored -> AppendMeasurements -> RecordScored) is not atomic. Concurrent processes could produce duplicate JSONL rows. Acknowledged in test comments; CI runs once per job.
fullsend-ai-review
Bot
added
requires-manual-review
|
🤖 Finished Review · ✅ Success · Started 1:20 AM UTC · Completed 1:41 AM UTC Commit: |
3 participants
Summary
Introduce eval measurements: fail-open same-job scoring of wild agent OTEL traces (
fullsend eval-measure), writing portableeval-measurements.jsonlbeside telemetry when at least one new score is produced. First scorer istrace_fitness(EM-001). Decision recorded in ADR 0087.Companion default-policy PR: fullsend-ai/agents#722 (manifests under
eval/measurements/).Activation is two-step: merge agents#722 and cut a
v0.x.yrelease that re-points floatingagents@v0. Until that release lands, managed GHA/GitLab measure steps stay provisional (clean skip when the remote manifest is missing). Tracking: #6384. LocalFULLSEND_DIRoverrides work today.Ownership (please read)
internal/evalmeasure/)FULLSEND_DIRfullsend-ai/agents@v0when no local file exists — installs do not copy manifests to score stock agents.${FULLSEND_DIR}/eval/measurements/${AGENT}.yamlis override / opt-out / custom-agent only.eval-measureis the released binary that readsrun-telemetry.jsonl(which fullsend writes). Agents is content/policy, not that binary.assert:→ fullsend PR; new id / enable / thresholds on an existing scorer for a stock agent → agents-only; org-specific policy → local override.Tool-agnostic export
Core does not pick an observability product. Scores land in local
eval-measurements.jsonl. Remote score export (when implemented) reuses the sameOTEL_EXPORTER_OTLP_*path as ADR 0050. No vendor Assessments adapters orMLFLOW_*(or similar) wiring in managed workflows.Related
agents@v0cut: #6384Changes
v0activation, declarative sketch, planned L3 content scorers), glossary / tracing cross-linksinternal/evalmeasureparser +trace_fitness+ local JSONL/ledgerfullsend eval-measureCLI; fail-open post-step inaction.ymland GitLabfullsend-agent.ymlFULLSEND_DIRthen SHA-pinnedagents@v0fetch (--offlinesupported)agent-<name>-…runDirs; EM-001 skips incomplete/runner-health traces (no agent span, missing rootrunspan, pre-script skip)GITLAB_ISSUE_URL; keep hostoutput/out of sandbox tarball + git excludeTesting
go test ./internal/evalmeasure/(+ focused CLI / sandbox / scaffold tests)FULLSEND_DIRmanifest path produceseval-measurements.jsonlagents@v0after merge + release cut (agents#722, #6384)Optional dogfood (outside core): post-process the portable JSONL into an org-chosen backend. Example MLflow UI showing
trace_fitnessassessments — not coreMLFLOW_*wiring:Checklist
Notes for reviewers
--registry(path to the YAML); rename to “manifest” is follow-up.v0(Track agents@v0 release cut for eval measurement manifests #6384).