Add opt-in/fail-closed exception to effective_on guidance in AGENTS.md

[fullsend-ai-retro]

What happened

On PR #1787, the fullsend-ai-review bot flagged a medium-severity [missing-effective-on] finding on the new signature_verification warn rule at policy/release/sbom/sbom.rego. The bot cited AGENTS.md lines 92-98, which state: "New deny/warn rules MUST include an effective_on date in their METADATA annotation to provide a migration window before enforcement begins."

The PR author (st3penta) explained in the PR body and review responses that effective_on was deliberately omitted: the rule is opt-in (only fires when a user explicitly configures the sbom entry in signing_identities rule data) and fail-closed (produces no output when unconfigured). Builds without that configuration never trigger the rule, so there is no migration window to grant, and users who opt in want failures surfaced immediately.

The current AGENTS.md guidance was added via issue #1760, which originated from a retro on PR #1736 where effective_on was missing on mandatory enforcement rules. However, #1760's implementation applied the requirement as a blanket "MUST" with no exception for opt-in/fail-closed rules, creating a false positive on this PR.

What could go better

The AGENTS.md guidance at lines 92-98 is too broad. The effective_on convention exists to prevent new mandatory rules from breaking existing builds on deployment. But rules that are both opt-in (require explicit user configuration to activate) and fail-closed (produce no output when unconfigured) cannot break existing builds by definition — unconfigured builds are completely unaffected. Requiring effective_on on such rules provides no safety benefit and creates friction when authors must justify a deliberate omission against a blanket "MUST" in review.

Confidence: High. The PR author's reasoning is sound and well-documented. The signature_verification rule only fires when a user adds an sbom entry to signing_identities rule data AND a signature check then fails. Without that config, the rule is invisible. This pattern will recur as more opt-in verification rules are added (e.g., for other artifact types), so the guidance should be refined now rather than requiring case-by-case justification on each future PR.

Proposed change

Amend the effective_on checklist item in AGENTS.md (lines 92-98) to add an explicit exception clause for opt-in/fail-closed rules. The updated text should:

1. Keep the existing "MUST" for mandatory enforcement rules (deny/warn rules that fire by default or affect all builds without explicit opt-in).
2. Add an exception: Rules that are both opt-in (require explicit user configuration in rule data to activate) and fail-closed (produce no output when unconfigured) may omit effective_on, since unconfigured builds are unaffected and users who opt in expect immediate enforcement.
3. Require that when effective_on is omitted for this reason, the PR description documents the rationale (as was done on PR Verify SBOM signatures from OCI referrers and tag refs #1787).

Example amended text after the existing guidance:

Exception: Rules that are both opt-in (require explicit user configuration to activate) and fail-closed (produce no output when unconfigured) may omit effective_on, since unconfigured builds are unaffected. When omitting effective_on for this reason, document the rationale in the PR description.

Validation criteria

1. The next review agent run on a PR that adds an opt-in/fail-closed rule without effective_on should not flag a missing-effective-on finding, provided the PR description documents the rationale.
2. The review agent should still flag missing effective_on on mandatory enforcement rules (deny/warn rules that fire by default).
3. Verify by checking the next 3 PRs that add new deny/warn rules to confirm the guidance produces correct review behavior for both mandatory and opt-in rules.

---

Generated by retro agent from #1787

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 7:39 AM UTC · Completed 7:42 AM UTC

Commit: 87c4a29 · View workflow run →

[fullsend-ai-triage]

Note: PR #1761 (for issue #1760) is still open and also modifies AGENTS.md in the same review checklist section. A developer working on this issue should coordinate with #1761 to avoid merge conflicts — ideally this amendment lands on top of or alongside #1761's changes.

Proposed test case

Scenario 1 — Mandatory rule without effective_on (should still be flagged):
  Given a new deny rule that fires by default on all builds
  When the rule omits effective_on from METADATA
  Then the review agent flags missing-effective-on

Scenario 2 — Opt-in/fail-closed rule without effective_on (should NOT be flagged):
  Given a new warn rule that requires explicit rule data config to activate
  And the rule produces no output when unconfigured
  When the rule omits effective_on and the PR description documents the rationale
  Then the review agent does NOT flag missing-effective-on

Scenario 3 — Opt-in but NOT fail-closed (should still be flagged):
  Given a new rule that requires config to activate but produces default output when unconfigured
  When the rule omits effective_on
  Then the review agent flags missing-effective-on

---

Next steps:

• /fs-code — agent creates a PR to implement this issue
• /fs-code <your instruction> — agent implements with your specific guidance

[fullsend-ai-coder]

🤖 Finished Code · ✅ Success · Started 7:44 AM UTC · Completed 7:49 AM UTC

Commit: 87c4a29 · View workflow run →