What happened
PR #7 added a Python-based PR type collector to a repo whose design.md documents a "Shell + gh + jq over Python" architecture decision. The review agent flagged this as an architecture-deviation finding in all 3 review iterations (R1 at 12:05, R2 at 19:50, R3 at 01:55 on Aug 13-14). The PR simultaneously amended design.md to document the Python exception, which the reviewer correctly identified as "self-authorizing" — the PR was both creating and justifying its own exception.
The repo has no AGENTS.md, no CLAUDE.md, and no .fullsend directory. The only repo-specific context available to agents is design.md and README.md. The review agent read design.md but had no pre-existing agent-facing guidance about accepted technologies, testing expectations, or data model conventions.
Additionally, the review agent's harness config defines REVIEW_PROTECTED_PATHS with a generic list (including .github/, scripts/), but this repo's scripts/ directory contains the core application logic (collectors), not just CI infrastructure. The protected-path HIGH finding fired on all 3 reviews because 6 files under .github/ and scripts/ were modified without a linked issue.
What could go better
An AGENTS.md file would give agents repo-specific context before they start work. Confidence: high — the architecture-deviation finding appeared in all 3 reviews and was the most persistent medium-severity finding. With pre-existing documentation of accepted patterns, the review agent would not need to flag a known exception.
Specifically:
- The Python exception is now documented in design.md (merged with this PR), but future PRs adding or modifying Python collectors may still trigger architecture-deviation findings if the review agent reads the Key Decisions section ("Shell + gh + jq over Python") before reaching the Dashboard evolution section that documents the exception. An AGENTS.md with explicit guidance would be more discoverable.
- The
scripts/ directory in this repo is application code (collectors), not CI infrastructure. The default REVIEW_PROTECTED_PATHS treats it as governance/infrastructure, causing every PR that modifies collectors to require human approval even for trivial changes.
- The repo has no testing conventions documented for agents, which contributed to the test-adequacy LOW finding in R3 asking for additional test coverage.
Uncertainty: it is possible that the review agent would still flag the architecture deviation even with AGENTS.md guidance, since the pr-review skill uses independent sub-agents that may not all read AGENTS.md. However, the harness injects AGENTS.md into the sandbox, making it available to all sub-agents that follow the file-reading instructions.
Proposed change
Create an AGENTS.md file in the repository root with the following sections:
-
Architecture decisions: Document that Python is accepted for complex collectors requiring cross-API orchestration (referencing the design.md "Python exception" section). State that shell + gh + jq remains the default for simple collectors.
-
Repository structure: Explain that scripts/ contains application logic (collectors and helpers), not CI infrastructure. Note that docs/ contains both the GitHub Pages dashboard and CSV data files.
-
Testing conventions: Document that Python collectors should have unit tests in scripts/test_*.py. Shell scripts are tested via manual verification.
-
Data model: Document the append-only CSV pattern (one row per repo per day), the two-file pattern (aggregates CSV + details CSV), and the idempotent-write pattern (skip existing (date, repo) pairs).
-
Known architecture exceptions: Explicitly list accepted deviations from the original design.md decisions so agents don't re-flag them.
Also create a CLAUDE.md that points to AGENTS.md (matching the pattern used in the fullsend-ai/agents repo).
Validation criteria
On the next 3 PRs to fullsend-ai/metrics that modify Python collector files, the review agent should not raise architecture-deviation findings about Python usage. The AGENTS.md should be readable by the review agent (verify it appears in the sandbox at /sandbox/workspace/target-repo/AGENTS.md during review runs).
Generated by retro agent from #7
What happened
PR #7 added a Python-based PR type collector to a repo whose design.md documents a "Shell + gh + jq over Python" architecture decision. The review agent flagged this as an architecture-deviation finding in all 3 review iterations (R1 at 12:05, R2 at 19:50, R3 at 01:55 on Aug 13-14). The PR simultaneously amended design.md to document the Python exception, which the reviewer correctly identified as "self-authorizing" — the PR was both creating and justifying its own exception.
The repo has no AGENTS.md, no CLAUDE.md, and no .fullsend directory. The only repo-specific context available to agents is design.md and README.md. The review agent read design.md but had no pre-existing agent-facing guidance about accepted technologies, testing expectations, or data model conventions.
Additionally, the review agent's harness config defines REVIEW_PROTECTED_PATHS with a generic list (including
.github/,scripts/), but this repo'sscripts/directory contains the core application logic (collectors), not just CI infrastructure. The protected-path HIGH finding fired on all 3 reviews because 6 files under.github/andscripts/were modified without a linked issue.What could go better
An AGENTS.md file would give agents repo-specific context before they start work. Confidence: high — the architecture-deviation finding appeared in all 3 reviews and was the most persistent medium-severity finding. With pre-existing documentation of accepted patterns, the review agent would not need to flag a known exception.
Specifically:
scripts/directory in this repo is application code (collectors), not CI infrastructure. The default REVIEW_PROTECTED_PATHS treats it as governance/infrastructure, causing every PR that modifies collectors to require human approval even for trivial changes.Uncertainty: it is possible that the review agent would still flag the architecture deviation even with AGENTS.md guidance, since the pr-review skill uses independent sub-agents that may not all read AGENTS.md. However, the harness injects AGENTS.md into the sandbox, making it available to all sub-agents that follow the file-reading instructions.
Proposed change
Create an
AGENTS.mdfile in the repository root with the following sections:Architecture decisions: Document that Python is accepted for complex collectors requiring cross-API orchestration (referencing the design.md "Python exception" section). State that shell + gh + jq remains the default for simple collectors.
Repository structure: Explain that
scripts/contains application logic (collectors and helpers), not CI infrastructure. Note thatdocs/contains both the GitHub Pages dashboard and CSV data files.Testing conventions: Document that Python collectors should have unit tests in
scripts/test_*.py. Shell scripts are tested via manual verification.Data model: Document the append-only CSV pattern (one row per repo per day), the two-file pattern (aggregates CSV + details CSV), and the idempotent-write pattern (skip existing (date, repo) pairs).
Known architecture exceptions: Explicitly list accepted deviations from the original design.md decisions so agents don't re-flag them.
Also create a
CLAUDE.mdthat points to AGENTS.md (matching the pattern used in the fullsend-ai/agents repo).Validation criteria
On the next 3 PRs to fullsend-ai/metrics that modify Python collector files, the review agent should not raise architecture-deviation findings about Python usage. The AGENTS.md should be readable by the review agent (verify it appears in the sandbox at /sandbox/workspace/target-repo/AGENTS.md during review runs).
Generated by retro agent from #7