Skip to content

Harden Dependabot Update Planner issues for atomic, reviewable execution #12517

Description

@mnkiefer

Problem

The latest live Dependabot Update Planner run created github/gh-aw#61573. Assigning Copilot produced github/gh-aw#61574, but review uncovered several problems that should have been prevented by the generated issue.

Findings

  1. The issue was not executable as assigned

    • The Dependabot orchestrator, not an assigned coding worker, must derive atomic, independently mergeable work units from manifest and lockfile resolution boundaries, canonical ownership and generated outputs, migration and security risk, validation scope, and rollback scope.
    • One unassigned parent/campaign issue should retain the inventory and lifecycle view. It is not a coding assignment.
    • An existing Dependabot PR should be reused directly as a work unit whenever it already has the correct boundary; the planner must not create a duplicate subissue for every Dependabot PR.
    • Create an independently assignable subissue only when additional implementation, consolidation of compatible updates, migration work, blocked/deferred lifecycle tracking, or other durable coordination is required.
    • Dispatch at most one worker per work unit, and permit each worker to produce or repair at most one PR. Workers do not repartition or discover other work.
    • High-risk major or security-sensitive units remain isolated and require the stated human review/gate.
  2. The dependency inventory was inconsistent

    • The issue summary claimed 15 updates: 5 Actions, 6 Go, and 4 npm.
    • The checklist contained 16 updates: 5 Actions, 4 Go, and 7 npm.
    • Counts must be derived from the final structured inventory rather than generated independently.
  3. Major-version migration requirements were incomplete

    • actions/upload-pages-artifact v4+ excludes hidden files by default.
    • The repository intentionally publishes docs/public/.well-known/ai.txt.
    • The upgrade therefore required include-hidden-files: true.
    • The planner should inspect release notes and repository-specific behavior, then state concrete migration invariants and validation steps.
  4. The canonical action-pin source was not identified

    • A stale literal setup-node SHA existed in pkg/workflow/maintenance_workflow_yaml_jobs.go.
    • The correct long-term fix is to use the shared getActionPin mechanism, not update another literal SHA.
    • The planner should identify canonical ownership, generated consumers, and exceptional duplicated pins before prescribing edits.
  5. Lockfile resolution exceeded the reviewed targets

    • @types/node resolved to 26.6.1 instead of the prescribed 26.5.1.
    • @primer/octicons resolved to 19.37.0 instead of 19.36.0.
    • Exact target versions and permitted transitive changes must be explicit.
    • Validation should reject lockfile drift beyond the reviewed Dependabot update.
  6. Candidate-version compatibility and security were not fully preflighted

    • Mermaid 12 was incompatible with astro-mermaid@2.1.0, whose peer range supports Mermaid 10 and 11.
    • Its candidate dependency graph also introduced known high-severity vulnerabilities.
    • “No current Dependabot alerts” is insufficient. The planner must inspect candidate peer dependencies and the resolved candidate graph before declaring an update actionable.
  7. Partial completion incorrectly closed the umbrella issue

    • Review narrowed the final PR to six npm updates across four files.
    • All GitHub Actions and Go changes were removed, and Mermaid was deferred.
    • The PR still used Fixes #61573, which closed the entire issue.
    • The issue checklist was not updated and no final validation report was posted.
    • The parent/campaign issue closes only after every work unit is completed, deferred, blocked, superseded, or otherwise in an explicit terminal state. Partial worker PRs must not close the parent; use Part of #... for partial work and reserve Fixes #... for complete fulfillment.
  8. PR metadata became stale after scope changed

    • The final PR title and body continued to claim Go and GitHub Actions updates after those changes were removed.
    • Agents must update the PR title, description, checklist, and validation report whenever review changes the scope.
    • PR title/body/checklist/validation evidence must remain synchronized with the current work-unit scope.

Required Changes

  • Have the Dependabot orchestrator derive work-unit boundaries—not the coding worker—from manifest/lockfile resolution boundaries, canonical ownership/generated outputs, migration and security risk, validation scope, and rollback scope; keep one unassigned parent/campaign issue for inventory and lifecycle tracking.
  • Reuse an existing Dependabot PR directly when it already has the correct atomic boundary; create an independently assignable subissue only for additional implementation, consolidation of compatible updates, migration work, blocked/deferred lifecycle tracking, or other durable coordination, and never create duplicate subissues for every Dependabot PR.
  • Dispatch at most one worker per work unit, allow each worker to produce or repair at most one PR, and prohibit workers from repartitioning or discovering other work; keep high-risk major/security-sensitive units isolated behind the stated human review/gate.
  • Close the parent only when every work unit is in an explicit terminal state: completed, deferred, blocked, superseded, or otherwise terminal; partial worker PRs must not close it.
  • Derive summary counts directly from the final dependency inventory.
  • Revalidate dependency state against the target repository’s exact HEAD before issuing work.
  • Distinguish updates that are already current, stale, superseded, blocked, or actionable.
  • Identify the canonical source of every dependency or action pin.
  • Identify generated consumers and prohibit direct edits to generated files.
  • Detect duplicated literal pins that should use an existing registry/helper.
  • Review major-version release notes and record concrete repository-specific migration requirements.
  • Inspect candidate peer dependencies and candidate lockfile vulnerabilities.
  • Freeze exact requested versions unless the issue explicitly authorizes a newer target.
  • Require generated lockfiles to match the reviewed versions and reject unrelated resolver drift.
  • Require validation of repository-specific artifacts and invariants, not only compilation.
  • Require Part of #... for partial batches and reserve Fixes #... for complete fulfillment.
  • Require agents to keep PR title, body, checklist, and validation results synchronized with the final diff.
  • Require unresolved or deferred updates to remain open and clearly reported.
  • Add focused tests covering these contracts.

Suggested Output Model

Each work unit (an existing Dependabot PR or a subissue) should contain:

  1. Unit type and link (existing Dependabot PR or subissue).
  2. Parent/campaign issue relation.
  3. One atomic dependency group with one review and rollback boundary.
  4. Exact source and target versions.
  5. Canonical manifest or pin source.
  6. Files expected to change.
  7. Files that must not be edited directly.
  8. Relevant release-note findings.
  9. Repository-specific invariants.
  10. Allowed lockfile and transitive dependency changes.
  11. Exact validation commands and expected evidence.
  12. Explicit blocked/deferred criteria.
  13. Worker dispatch (at most one worker and one PR).
  14. Terminal status and completion instructions for issue and PR metadata.

Regression Fixtures

Add fixtures based on github/gh-aw#61573 and github/gh-aw#61574 proving that the planner:

  • Derives boundaries in the orchestrator from resolution boundaries, ownership/generated outputs, risk, validation, and rollback scope, rather than delegating partitioning to a worker.
  • Reuses correctly bounded existing Dependabot PRs as work units without creating duplicate subissues.
  • Creates subissues only when additional implementation, compatible-update consolidation, migration, blocked/deferred tracking, or other durable coordination requires one.
  • Does not combine isolated GitHub Actions, crypto-sensitive Go updates, and npm clusters into one assignable work unit, and cannot let one worker combine multiple units.
  • Produces consistent inventory counts.
  • Detects the hidden-file requirement for upload-pages-artifact.
  • Identifies shared action-pin ownership and duplicated literals.
  • Rejects lockfiles resolving beyond exact reviewed targets.
  • Defers incompatible or newly vulnerable candidate versions.
  • Cannot close the parent through a partial implementation; parent closure waits until every work unit has an explicit terminal state.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions