feat(workflows): extract harness-run into reusable workflow - #6356
feat(workflows): extract harness-run into reusable workflow#6356ralphbean wants to merge 10 commits into
Conversation
Extracts the harness-run job from reusable-dispatch.yml into a new reusable-harness-run.yml workflow that can be invoked directly from user repos. This enables custom pollers to invoke harness agents without going through the full dispatch flow. Fixes #6347 Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
ralphbean
added
the
fullsend-fix
|
🤖 Review · Commit: |
PR Summary by QodoExtract harness execution into a reusable workflow
AI Description
Diagram
High-Level Assessment
Files changed (2)
|
|
🤖 Finished Review · ✅ Success · Started 1:02 AM UTC · Completed 1:21 AM UTC Commit: |
Code Review by Qodo
1.
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
ReviewFindingsMedium
Low
Previous runReviewFindingsHigh
Medium
Low
Next steps:
Previous run (2)ReviewFindingsMedium
Low
Next steps:
Previous run (3)ReviewFindingsHigh
Medium
Low
Next steps:
Previous run (4)ReviewFindingsHigh
Medium
Low
Next steps:
Previous run (5)ReviewFindingsMedium
Low
Next steps:
Previous run (6)ReviewFindingsMedium
Low
Previous run (7)ReviewFindingsHigh
Medium
Low
Labels: PR modifies .github/workflows/ dispatch and harness execution infrastructure Next steps:
|
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
fullsend-ai-review
Bot
added
component/ci
|
🤖 Finished Fix · ❌ Failure · Started 1:22 AM UTC · Completed 1:28 AM UTC Commit: |
|
The fix agent completed, but the post-fix script failed before finishing.
Workflow run: https://github.com/fullsend-ai/.fullsend/actions/runs/32204629702 Details: |
Update vendormanifest and workflow alignment tests to account for harness-run being extracted into reusable-harness-run.yml: - Add reusable-harness-run.yml to vendored reusable workflows list - Update OTEL headers/variables tests to check reusable-harness-run.yml separately instead of the inline harness agent step - Update PR head SHA passthrough test to verify the workflow_call to reusable-harness-run.yml instead of the inline step Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
008072d to
f060999
Compare
|
🤖 Review · ❌ Terminated · Started 2:25 AM UTC · Ended 2:41 AM UTC Commit: |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Superseded by updated review
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
fullsend-ai-review
Bot
added
the
requires-manual-review
|
🤖 Finished Review · ✅ Success · Started 2:25 AM UTC · Completed 2:41 AM UTC Commit: |
waynesun09
left a comment
waynesun09
left a comment
There was a problem hiding this comment.
Review findings
One HIGH and three MEDIUM findings. Three are inline; the one below could not be anchored inline because the line it concerns sits outside this PR's diff hunks.
MEDIUM — New caller→callee hop escapes TestWorkflowCallInputAlignment, and its regex cannot match the hyphenated filename
internal/scaffold/workflow_call_alignment_test.go:177
Two compounding gaps, both verified at head:
- The pattern
reusable-[a-z]+\.yml(reusableWorkflowRef, line 177) cannot matchreusable-harness-run.yml— afterreusable-,[a-z]+consumesharnessand then requires\.yml, but the next char is-; no backtracking recovers. - The
pairslist (lines 220-226) contains only the six scaffold thin callers and was not extended with the newreusable-dispatch.ymlharness-run→reusable-harness-run.ymlrelationship. The adjacent comment "reusable-dispatch.yml stage jobs are no longer validated here (ADR 62: stages inlined, no external uses:)" is now factually false, since this PR reintroduces exactly such auses:.
So the one automated guard against caller/callee input+secret drift does not cover the hop this PR creates — 6 inputs and 4 secrets are now threaded by hand across it — and even adding a pair would silently no-op because of the regex. I verified the pair is aligned today (the callee declares matrix/install_mode/mint_url/gcp_region/fullsend_version/runner_image and the 4 secrets; the caller passes exactly those); the risk is unguarded future drift. docs/contributing/workflow-contracts.md:14 explicitly instructs contributors to extend these tests when adding a hop.
Suggestion: Widen the pattern to reusable-[a-z0-9-]+\.yml, add a callerPair entry {"reusable-dispatch.yml", loadRepoFile(".github/workflows/reusable-dispatch.yml"), "harness-run"} (the callerWorkflow YAML struct already parses uses/with/secrets), delete/correct the now-false "no external uses:" comment, and add a regression assertion that the regex matches reusable-harness-run.yml so the next hyphenated workflow is not silently skipped.
| EVENT_PAYLOAD: ${{ matrix.event_payload }} | ||
| run: | | ||
| set -euo pipefail | ||
| URL=$(printf '%s' "${EVENT_PAYLOAD}" | jq -r '.issue.html_url // .pull_request.html_url // empty') |
There was a problem hiding this comment.
MEDIUM — Motivating poller use case is structurally blocked: the extracted body hard-codes GitHub event/identifier shapes
The PR's stated motivation (and #6347's) is letting a custom poller invoke harness agents directly, and the second test-plan box is checked against run 32202963756. That run failed at "Run harness agent" with ERROR: ISSUE_URL does not match expected pattern: https://stage-redhat.atlassian.net/browse/KONFLUX-15169 plus github api: 404 Not Found status posts against "issue" #6580022 (a Jira id).
That is structural, not incidental: line 192 derives GITHUB_ISSUE_URL from jq -r '.issue.html_url // .pull_request.html_url // empty' and hard-fails when neither is present, and lines 218-224 pass matrix.source_repo/status_repo/status_number to the action as GitHub identifiers — precisely the assumptions a fullsend poll --input-driver jira-poll caller cannot satisfy. So no linked run demonstrates the new capability working.
For accuracy on the other checkbox: run 32203213291 shows dispatch / Harness run / Harness run (${{ matrix.agent }}) as skipped with the literal unexpanded matrix expression, i.e. the harness path never executed there either. The real dispatch-path evidence is the PR's own behaviour run (test-repo-03, run 32208540212), where steps 1-11 including Mint agent token and Setup GCP all succeeded and step 12 died on a podman-download flake (curl: (35)) — that does validate the extraction and its permission inheritance.
Suggestion: Either scope the PR and #6347 explicitly to GitHub-shaped event payloads and file the Jira/non-GitHub payload gap as follow-up, or uncheck the direct-invocation box until a run demonstrates it end to end. Also relink the dispatch-path checkbox to a run where the harness job actually executed (e.g. 32208540212) rather than 32203213291, where it was skipped.
There was a problem hiding this comment.
We can come back to this after a few iterations - but this really did work. I may just need to update the example I linked to.
The one thing that's not yet working is how to get the JIRA_TOKEN through from the calling repo to the harness-run workflow and the composite action (unrelated to your thread here, other than its the one thing (I think) that's left to get this working.)
There was a problem hiding this comment.
I've really gotten pre-scripts to work (see here: https://github.com/ralphbean/jira-triage-test/actions/runs/32281375216/job/96160837024 ) but am now hitting unrelated sandbox creation issues.
There was a problem hiding this comment.
OK - yes, this works.
- example run from a jira-poll workflow: https://github.com/ralphbean/jira-triage-test/actions/runs/32496078912/job/96814799790 with direct usage of reusable-harness-run
- example run from the same repo with reusable-dispatch (proving it didn't get broken in the process): https://github.com/ralphbean/jira-triage-test/actions/runs/32413923978
Declare JIRA_TOKEN, JIRA_USER_EMAIL, and JIRA_BASE_URL as optional secrets in both reusable-dispatch.yml and reusable-harness-run.yml. Expose them as env vars in the harness agent step so they're available to pre-scripts and the fullsend run environment. This enables JIRA-based agents (like triage) to authenticate with Jira APIs without requiring bespoke workflow files. For issue #6359. Assisted-by: Claude claude-sonnet-4-5@20250929 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
|
🤖 Finished Review · ✅ Success · Started 5:14 PM UTC · Completed 5:34 PM UTC Commit: |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
fullsend-ai-review
Bot
removed
the
requires-manual-review
- Fix OTEL secret-threading test to extract step section instead of matching whole file content (which includes header comment examples) - Fix OTEL variable-forwarding test with same section extraction - Remove workflow_sha input and use job.workflow_sha built-in context - Move permissions block from my-poller to harness-run in example - Remove workflow_sha: v0 from example (forces source build vs release) Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
|
🤖 Finished Review · ✅ Success · Started 5:41 PM UTC · Completed 6:03 PM UTC Commit: |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
🤖 Finished Fix · ❌ Failure · Started 6:04 PM UTC · Completed 6:12 PM UTC Commit: |
|
The fix agent completed, but the post-fix script failed before finishing. Workflow run: https://github.com/fullsend-ai/fullsend/actions/runs/32401038626 Details: Please check the workflow logs for full details and retry with |
The workflow_sha input was removed from reusable-harness-run.yml in commit 9393f44 (it now uses job.workflow_sha directly), but reusable-dispatch.yml was still trying to pass it. This caused startup_failure when calling the workflow. Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
|
🤖 Finished Review · ✅ Success · Started 6:22 PM UTC · Completed 6:43 PM UTC Commit: |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
🤖 Finished Fix · ❌ Failure · Started 6:45 PM UTC · Completed 6:53 PM UTC Commit: |
|
The fix agent completed, but the post-fix script failed before finishing.
Workflow run: https://github.com/fullsend-ai/fullsend/actions/runs/32404743283 Details: |
…tory The workflow_repository input had the same issue as workflow_sha - it was being passed as github.repository from reusable-dispatch.yml, which refers to the calling repository (e.g., ralphbean/jira-triage-test), not the repository containing the workflow (fullsend-ai/fullsend). This caused checkout failures when trying to fetch a SHA from the fullsend repo while checking out the jira-triage-test repo. Solution: Remove the workflow_repository input and use job.workflow_repository directly in reusable-harness-run.yml, just like we did for workflow_sha. Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
|
🤖 Finished Review · ✅ Success · Started 8:01 PM UTC · Completed 8:23 PM UTC Commit: |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
🤖 Finished Fix · ✅ Success · Started 8:24 PM UTC · Completed 8:31 PM UTC Commit: |
Update the reusableWorkflowRef regex from [a-z]+ to [a-z-]+ so it matches hyphenated workflow names like reusable-harness-run.yml. Without this, TestWorkflowCallInputAlignment cannot be extended to cover the new workflow. Addresses review feedback on #6356
🔧 Fix agent — iteration 1 (bot-triggered)Fixed 1 of 3 review findings: updated reusableWorkflowRef regex to allow hyphens. Disagreed with 2 findings: the authorization-bypass claim is factually incorrect (code already uses job.workflow_repository), and the secret-threading mismatch is a documented known inconsistency out of scope for this PR. Fixed (1):
Disagreed (2):
Tests: passedNext steps:
|
|
🤖 Finished Review · ✅ Success · Started 8:33 PM UTC · Completed 8:51 PM UTC Commit: |
Superseded by updated review
fullsend-ai-review
Bot
added
the
requires-manual-review
waynesun09
left a comment
waynesun09
left a comment
There was a problem hiding this comment.
Review sweep findings (2 unique after deduping against existing review threads on this PR).
| # with: | ||
| # matrix: ${{ needs.my-poller.outputs.matrix }} | ||
| # mint_url: ${{ vars.FULLSEND_MINT_URL }} | ||
| # gcp_region: ${{ vars.GCP_REGION }} |
There was a problem hiding this comment.
HIGH — Documented caller example uses the wrong GCP region var and omits Jira secret forwarding
The header comment's "Minimal caller example" sets gcp_region: ${{ vars.GCP_REGION }} (this line), but the repo's own actual caller — internal/scaffold/fullsend-repo/templates/shim-per-repo.yaml line 51 — uses vars.FULLSEND_GCP_REGION. A custom poller following the documented example verbatim will pass an unset/empty var for a required input, silently breaking GCP WIF auth. The same example's secrets: block (lines 43-47) also does not forward JIRA_TOKEN/JIRA_USER_EMAIL even though the workflow declares them and the surrounding docs describe Jira usage, so a copy-paste caller gets no working Jira integration either. Confirmed by diffing the header example against the real shim template and the workflow_call.secrets declarations at head cc0486c.
Suggestion: Fix the example to use vars.FULLSEND_GCP_REGION (matching the real shim), and add JIRA_TOKEN/JIRA_USER_EMAIL secret forwards to the example so it matches the actual workflow_call contract.
| # id-token: write | ||
| # issues: write | ||
| # pull-requests: write | ||
| # uses: fullsend-ai/fullsend/.github/workflows/reusable-harness-run.yml@v0 |
There was a problem hiding this comment.
HIGH — Reintroduces the ADR-0062 @v0 version-skew pattern for a new, less-controlled external-caller audience
docs/ADRs/0062-dispatch-version-skew.md (Accepted) identifies hardcoded reusable-<stage>.yml@v0 references as the "version skew" problem, rejects re-introducing per-repo uses: refs, and mandates inlining stage workflows into reusable-dispatch.yml specifically to remove the extra workflow_call hop. This PR's header comment instructs a brand-new class of external caller — custom pollers in repos not managed by fullsend install — to hardcode uses: fullsend-ai/fullsend/.github/workflows/reusable-harness-run.yml@v0 (this line), with no update/versioning mechanism proposed for those repos to track changes, i.e. the same version-skew failure mode ADR 0062 eliminated, relocated to a less-controlled, unmanaged audience. This is not acknowledged anywhere in the PR body, the new file, or ADR 0062 itself. Additionally, reusable-dispatch.yml's own header comment ("Nesting: 2 levels of workflow_call (previously 3 before inlining)", line 16) is now stale/incorrect since this PR reintroduces a third hop for the harness-run path specifically.
Suggestion: Amend or supersede ADR 0062 to explicitly scope its rule to the per-repo dispatch flow and document why direct external invocation is a separate risk category with its own versioning story (e.g. real vMAJOR.MINOR.PATCH tags + bump guidance instead of @v0), and fix the stale nesting-depth comment in reusable-dispatch.yml.
There was a problem hiding this comment.
Oh, this is a problem. I don't want to amend or supersede ADR 62 just to say "this is ok". This is a real problem. People who pin reusable dispatch to a particular version will still "float" forward in their unpinned indirect reference to reusable-harness-run.
|
Putting this one into draft while I think about how to resolve the version skew problem. IMO that's a blocker. |
2 participants
Summary
harness-runjob fromreusable-dispatch.ymlinto a newreusable-harness-run.ymlworkflowreusable-dispatch.ymlremains unchangedChanges
New file:
.github/workflows/reusable-harness-run.ymlmatrixinput (JSON string fromfullsend dispatch)Modified:
.github/workflows/reusable-dispatch.ymlharness-runjob to call the new reusable workflow withuses:harness-dispatchTest plan
@extract-harness-runto confirm run and commitFixes #6347
🤖 Generated with Claude Code