What happened
On PR #1816, the review agent (run 32211215873) approved and applied ready-for-merge to a PR that added a vendor-specific PNC build type to the global defaults map in rule_data.rego. Ten hours later, human reviewer st3penta caught the design issue: adding a vendor-specific type to global defaults enables it for all Conforma users, and the correct approach is per-deployment custom rule_data. The agent had contextual signals (all 5 existing allowed_provenance_build_types entries are Tekton-specific URLs, the new entry uses project-ncl.github.io), but AGENTS.md provides no guidance about the rule_data defaults mechanism, the 4-tier priority hierarchy (rule_data__configuration__ > rule_data_custom > rule_data > defaults), or criteria for what belongs in global defaults versus per-deployment configuration.
What could go better
The review agent (and any new human contributor) lacks the domain context to evaluate changes to the hardcoded defaults map in rule_data.rego. The rule_data.get() function implements a 4-tier replacement-based priority hierarchy, but this architecture and its implications are undocumented outside the code itself. The design/ directory has a doc about the object.union merge pattern but nothing about inclusion criteria for the defaults map. Without explicit guidance, the agent cannot distinguish between a universally appropriate default (like Tekton build types, which this project was built for) and a vendor-specific configuration that should be scoped to individual deployments. I am confident this documentation gap was the root cause of the missed finding, since the agent demonstrated it can detect other kinds of issues (formatting, test patterns, metadata staleness) when it has context to evaluate them.
Proposed change
Add a section to AGENTS.md under the existing "Architecture" heading that documents:
-
The rule_data priority hierarchy: Explain that rule_data.get(key) resolves values through 4 tiers (ECP configuration > custom data source > default data source > hardcoded defaults in rule_data.rego), using replacement semantics (first match wins entirely, no merging across tiers).
-
Inclusion criteria for the hardcoded defaults map: Entries in defaults apply to every Conforma deployment. Only values that are universally appropriate belong there (e.g., Tekton build types, which are the standard this project targets). Vendor-specific, deployment-specific, or organization-specific values should be configured via custom rule_data or rule_data__configuration__ at the deployment level.
-
Review checklist item: When reviewing PRs that modify the defaults map in rule_data.rego, ask: "Is this value universally appropriate for all Conforma users, or is it specific to a particular vendor, organization, or deployment?" If vendor/deployment-specific, recommend the per-deployment custom rule_data mechanism instead.
Reference the existing conforma.dev/docs/cli/configuration.html#_data_sources documentation for how users configure custom rule data.
Validation criteria
The next review agent run on a PR that adds a vendor-specific or deployment-specific entry to the defaults map in rule_data.rego should flag the change with a finding recommending per-deployment configuration instead of global defaults. Verify by checking the next 3 PRs that modify rule_data.rego defaults — the review agent should reference the new AGENTS.md guidance when assessing whether an addition is universally appropriate.
Generated by retro agent from #1816
What happened
On PR #1816, the review agent (run 32211215873) approved and applied
ready-for-mergeto a PR that added a vendor-specific PNC build type to the globaldefaultsmap inrule_data.rego. Ten hours later, human reviewer st3penta caught the design issue: adding a vendor-specific type to global defaults enables it for all Conforma users, and the correct approach is per-deployment customrule_data. The agent had contextual signals (all 5 existingallowed_provenance_build_typesentries are Tekton-specific URLs, the new entry usesproject-ncl.github.io), butAGENTS.mdprovides no guidance about therule_datadefaults mechanism, the 4-tier priority hierarchy (rule_data__configuration__>rule_data_custom>rule_data>defaults), or criteria for what belongs in global defaults versus per-deployment configuration.What could go better
The review agent (and any new human contributor) lacks the domain context to evaluate changes to the hardcoded
defaultsmap inrule_data.rego. Therule_data.get()function implements a 4-tier replacement-based priority hierarchy, but this architecture and its implications are undocumented outside the code itself. Thedesign/directory has a doc about theobject.unionmerge pattern but nothing about inclusion criteria for the defaults map. Without explicit guidance, the agent cannot distinguish between a universally appropriate default (like Tekton build types, which this project was built for) and a vendor-specific configuration that should be scoped to individual deployments. I am confident this documentation gap was the root cause of the missed finding, since the agent demonstrated it can detect other kinds of issues (formatting, test patterns, metadata staleness) when it has context to evaluate them.Proposed change
Add a section to
AGENTS.mdunder the existing "Architecture" heading that documents:The
rule_datapriority hierarchy: Explain thatrule_data.get(key)resolves values through 4 tiers (ECP configuration > custom data source > default data source > hardcoded defaults inrule_data.rego), using replacement semantics (first match wins entirely, no merging across tiers).Inclusion criteria for the hardcoded
defaultsmap: Entries indefaultsapply to every Conforma deployment. Only values that are universally appropriate belong there (e.g., Tekton build types, which are the standard this project targets). Vendor-specific, deployment-specific, or organization-specific values should be configured via customrule_dataorrule_data__configuration__at the deployment level.Review checklist item: When reviewing PRs that modify the
defaultsmap inrule_data.rego, ask: "Is this value universally appropriate for all Conforma users, or is it specific to a particular vendor, organization, or deployment?" If vendor/deployment-specific, recommend the per-deployment customrule_datamechanism instead.Reference the existing
conforma.dev/docs/cli/configuration.html#_data_sourcesdocumentation for how users configure custom rule data.Validation criteria
The next review agent run on a PR that adds a vendor-specific or deployment-specific entry to the
defaultsmap inrule_data.regoshould flag the change with a finding recommending per-deployment configuration instead of global defaults. Verify by checking the next 3 PRs that modifyrule_data.regodefaults — the review agent should reference the new AGENTS.md guidance when assessing whether an addition is universally appropriate.Generated by retro agent from #1816