feat: collect human rework rate for bot-merged PRs

[waynesun09]

Context

Moved from fullsend-ai/fullsend#5516 / PR fullsend-ai/fullsend#5517 (closed — wrong repo). The goal is a trended trust metric for fullsend-ai's own agents, which is what this repo already does; a one-off script in the fullsend repo has no persistence, is not shipped to action users, and only fullsend developers can run it.

scripts/collect-rework.sh currently measures bot-on-bot rework and explicitly scopes out human rework. This fills that gap.

Proposal

Add a daily collector (e.g. scripts/collect-human-rework.sh on lib.sh, wired into .github/workflows/collect.yml) that, for each PR merged by a fullsend bot on fullsend + agents:

1. Records the files touched by the PR
2. Checks for human (non-bot, author.type == "User", non-null author) commits within N days (default 7) touching the same files
3. Persists per-day × repo counts (bot_prs_merged, bot_prs_reworked, rework_rate) plus a per-PR details CSV, and surfaces it on the dashboard

The reference implementation with tests is in fullsend-ai/fullsend#5517 (scripts/rework-rate.sh) — logic is reviewed and solid; it needs porting onto lib.sh + CSV persistence rather than recomputing from scratch each run. Known limitations (repo-wide file-overlap heuristic, pre-merge fix branches, PAT machine users counted as human) are documented in that PR's description.

Related

• feat(eval): add rework rate tracking for agent PRs fullsend#5516, chore(eval): add rework rate tracking script for agent PRs (#5516) fullsend#5517
• Define quality metrics for autonomous software factory fullsend#295 (roadmap: rework rate tracking)
• docs/problems/trustworthiness-evidence.md in fullsend

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 7:12 PM UTC · Completed 7:16 PM UTC

Commit: 7aa8587 · View workflow run →

[fullsend-ai-triage]

Workflow changes required

This feature requires adding a new step to .github/workflows/collect.yml. The code agent cannot modify workflow files under current permissions — a human will need to wire the new collector step into the workflow after the script is implemented.

Implementation notes

The existing collect-rework.sh provides a strong pattern to follow. Key differences from bot-on-bot rework:

• Scope: only bot-merged PRs on fullsend + agents (not all org repos)
• Detection: file-level overlap between bot PR and subsequent human commits, not repeated bot touches on the same item
• New CSV files: docs/human-rework.csv and docs/human-rework-details.csv (separate from existing rework CSVs)

The reference implementation in the upstream PR (fullsend-ai/fullsend#5517, now closed) has reviewed logic that should be ported onto lib.sh + CSV persistence rather than recomputing from scratch each run.

Proposed test case

Given a date where:
  - Bot merged PR #100 touching files [a.ts, b.ts]
  - Human committed to a.ts 3 days later
  - Bot merged PR #101 touching files [c.ts]
  - No human commits touched c.ts within 7 days

Expect:
  human-rework.csv: date,repo,2,1,0.500
  human-rework-details.csv: PR #100 flagged is_rework=true
  human-rework-details.csv: PR #101 flagged is_rework=false
  Re-running for same date: no duplicate rows

---

Labels: This is a new feature request for a metric collector, matching the 'enhancement' label.

---

Next steps:

• /fs-code — agent creates a PR to implement this issue
• /fs-code <your instruction> — agent implements with your specific guidance