bug: recurring git-sync fails entirely when a git branch maps to an already-merged (read-only) Infrahub branch

[minitriga]

Component

Git Integration

Infrahub version

Not specified by reporter (observed via the recurring git_repositories_sync flow; traceback below).

Current Behavior

The recurring git_repositories_sync flow records each branch's commit through update_commit_value → CoreRepositoryUpdate. When a git branch still present on the remote (here description-field) maps to an Infrahub branch that has already been merged (and is therefore read-only), the mutation is rejected:

Branch 'description-field' has been merged and is read-only. No modifications are allowed.

The resulting GraphQLError is not among the exceptions collect_pending_imports isolates per branch (RepositoryError, CommitNotFoundError, GitCommandError, ValueError), and none of the enclosing frames catch it either (sync_git_repo_with_origin_and_tag_on_failure and sync_repository_from_origin catch only RepositoryError/CommitNotFoundError; sync_remote_repositories has no per-repo guard). So a single merged-but-undeleted git branch:

• aborts collection for all other branches of the same repository that cycle, and
• propagates unhandled to the top of the flow, aborting the entire flow run — every repository iterated after the failing one is skipped that cycle.

Because this is the per-minute cron and the git branch persists until deleted, the failure recurs every cycle: git-sync is effectively frozen for the affected repository (and all later-ordered repositories) until the branch is removed from the remote.

Expected Behavior

A git branch that maps to a merged/read-only Infrahub branch should not break the recurring sync. Either the sync skips recording commits for merged/read-only branches, or the read-only-branch rejection is isolated per branch (like the other per-branch failures) so the remaining branches and repositories still sync.

Steps to Reproduce

1. Register a read-write CoreRepository and let it sync.
2. Create a working branch (e.g. description-field) that exists both on the git remote and as an Infrahub branch.
3. Merge that Infrahub branch (it becomes read-only) without deleting the corresponding branch on the git remote.
4. Let the per-minute git_repositories_sync flow run.
5. The flow fails with the GraphQLError above; sync for that repository (and any repository iterated after it) no longer progresses.

Additional Information

Traceback

infrahub_sdk.exceptions.GraphQLError: An error occurred while executing the GraphQL Query
 mutation ($repository_id: String!, $commit: String!) {
   CoreRepositoryUpdate(data: { id: $repository_id, commit: { is_protected: true, source: $repository_id, value: $commit } }) {
     ok
     object { commit { value } }
   }
 }
, [{'message': "Branch 'description-field' has been merged and is read-only. No modifications are allowed.",
   'locations': [{'line': 3, 'column': 9}], 'path': ['CoreRepositoryUpdate'],
   'extensions': {'code': 'UNDEFINED_ERROR', 'http_status': 400, 'data': {}}}]

  File ".../infrahub/git/tasks.py", sync_remote_repositories
  File ".../infrahub/git/tasks.py", sync_repository_from_origin
  File ".../infrahub/git/tasks.py", sync_git_repo_with_origin_and_tag_on_failure
  File ".../infrahub/git/sync.py", RepositorySyncer.sync -> collect_pending_imports
  File ".../infrahub/git/repository.py", collect_pending_imports -> update_commit_value
  File ".../infrahub/git/base.py", update_commit_value -> sdk.repository_update_commit

(Reporter's traceback line numbers correspond to an earlier release; the code path is unchanged on the current stable line: collect_pending_imports in backend/infrahub/git/repository.py calls update_commit_value, and the GraphQLError it raises falls outside the isolated exception tuple.)

References

• bug: recurring git-sync stalls permanently after a worker dies mid-submission (leaked deployment concurrency slot) #9925 — same symptom (recurring git-sync frozen) but a different cause (Prefect concurrency-slot leak, not an unhandled mutation error). Not a duplicate.
• Related area, distinct bugs: bug: recording a read-only repository's commit re-triggers an echo re-import that can auto-sync new commits #9827, bug: merge to non-main default_branch silently dropped on multi-worker pools #9568, bug: Git merge skipped if Proposed Change merged before repo sync completes #9499

[minitriga]

/bug-analyze

[opsmill-bug-pipeline]

✅ Bug analyst agent completed successfully!

[opsmill-bug-pipeline]

Root cause analysis for #9931

Issue: bug: recurring git-sync fails entirely when a git branch maps to an already-merged (read-only) Infrahub branch
Based on: 2a8e5c8229a6194fbd8be326dc405eaa9da9a529
Bug clarity: CLEAR
Code identification: RESOLVED

Root cause

When a git branch that maps to an already-merged (read-only) Infrahub branch is collected as a "new" branch, collect_pending_imports calls update_commit_value, whose GraphQLError ("Branch ... has been merged and is read-only") falls outside the per-branch exception tuple and every enclosing catch, so it aborts the whole repository sync and the entire flow.

Affected files

• backend/infrahub/git/repository.py -- lines 162-193 (collect_pending_imports, new-branches loop): the except (RepositoryError, CommitNotFoundError, GitCommandError, ValueError) at line 185 does not include infrahub_sdk.exceptions.GraphQLError, so the read-only rejection raised at line 180 (update_commit_value) escapes the per-branch isolation.
• backend/infrahub/git/base.py -- line 628-646 (update_commit_value): unconditionally calls sdk.repository_update_commit, which raises GraphQLError when the mapped Infrahub branch is read-only (merged/merging).
• backend/infrahub/git/sync.py -- lines 108-110 (RepositorySyncer.sync): collect_pending_imports is awaited under the lock but outside the per-branch try/except (which starts at line 113), so anything it raises propagates straight out of sync().
• backend/infrahub/git/tasks.py -- lines 232, 366 (sync_git_repo_with_origin_and_tag_on_failure, sync_repository_from_origin) catch only (RepositoryError, CommitNotFoundError); line 380 (sync_remote_repositories) has no per-repo guard. None of these catch GraphQLError, so it aborts the whole flow.

Explanation

When an Infrahub branch is merged it is retained as a read-only node (status MERGED), while the corresponding git branch persists on the remote until deleted. On a subsequent sync where the worker's local clone no longer tracks that branch, compare_local_remote (base.py:843) classifies it as a new branch. In the new-branches loop of collect_pending_imports (repository.py:162):

1. create_branch_in_graph raises GraphQLError containing "already exist"; this is specifically caught (line 171-174) and the existing branch is fetched via sdk.branch.get — returning BranchData with status == BranchStatus.MERGED.
2. Execution continues to update_commit_value (line 180), which calls sdk.repository_update_commit. The server rejects the write in backend/infrahub/branch/status_checker.py:8-12 (BranchStatusChecker.check_merge_status, raising BranchAlreadyMergedError), surfaced to the caller as an infrahub_sdk.exceptions.GraphQLError.
3. GraphQLError (subclass of the SDK's Error, unrelated to infrahub.exceptions.RepositoryError) is not in the tuple at line 185, so it is not isolated as a FailedImport. It propagates out of collect_pending_imports, out of sync() (raised outside the per-branch guard), and past every except (RepositoryError, CommitNotFoundError) frame in tasks.py, with no per-repo guard in sync_remote_repositories.

Net effect: the merged git branch aborts collection for all remaining branches of that repository and aborts the whole flow run, skipping every later-ordered repository. Because it is the per-minute cron and the git branch persists until deleted, git-sync is effectively frozen for the affected repository (and all repositories iterated after it) each cycle.

Fix strategy

Approach: Isolate the read-only/merged-branch rejection at branch granularity inside collect_pending_imports, and treat it as a benign skip rather than a failure. Recommended: in the new-branches loop, once the existing branch has been fetched in the "already exist" path (repository.py:174), inspect the returned BranchData.status and, when it is a read-only state (BranchStatus.MERGED / BranchStatus.MERGING), log at debug/info and continue before calling update_commit_value — skip recording a commit for a branch that can no longer accept writes. This reuses the BranchData.status field the SDK already returns (python_sdk/infrahub_sdk/branch.py:34) and mirrors the existing GraphQLError-message inspection pattern already used a few lines above for "already exist".

As a robustness backstop (not a substitute), also guard the write path: catch GraphQLError raised by update_commit_value and, only when its errors[0]["message"] indicates the branch is merged/read-only, isolate it (skip that branch) rather than propagate; re-raise any other GraphQLError.

Scope: Small, localized change in backend/infrahub/git/repository.py within collect_pending_imports (the new-branches loop). GraphQLError and RepositoryInternalStatus are already imported there; BranchStatus would be a new import from infrahub_sdk.branch. No changes to sync.py, base.py, or the mutation/status-checker logic are required for the core fix.

Do NOT:

• Do NOT simply add GraphQLError to the broad except (...) tuple at repository.py:185 and route it into failed_imports unconditionally. That both masks genuine graph-write failures for unrelated reasons and marks the merged branch as a permanent per-cycle failure — raise_if_branches_failed (repository.py:127) would then raise a RepositoryError and tag the repo with a sync error every single minute. A merged branch is expected, not a failure; skip it, don't fail it.
• Do NOT broaden the except (RepositoryError, CommitNotFoundError) blocks in tasks.py to swallow all exceptions, and do NOT "fix" this by wrapping the loop body in sync_remote_repositories with a blanket except Exception. The correct granularity is per-branch inside collect_pending_imports; broadening top-level catches hides unrelated repo/flow failures.
• Do NOT delete the git branch from the remote or otherwise mutate remote state as part of the sync.
• Do NOT change status_checker.py / the read-only branch mutation rejection — the server-side guard is correct; the bug is that the sync doesn't handle it.

Notes for downstream steps

• The merged-branch case reaches the new-branches loop specifically because the branch is missing from the worker's local clone (so compare_local_remote returns it as new) while the Infrahub branch node still exists (so create_branch_in_graph raises "already exist"). A reproduction should recreate that state: an Infrahub branch that has been merged (status MERGED), its git branch still present on the remote, and the worker treating it as a new branch.
• BranchStatusChecker.check_merge_status rejects both MERGED and MERGING, so the skip/isolation should cover both read-only statuses, not just MERGED.
• Verify the desired end state: after the fix, other branches of the same repo and all later-ordered repositories must still sync in the same cycle, and the merged branch should be skipped without marking the repository as errored/tagged each cycle.
• Existing component tests for this path live in backend/tests/component/git/test_git_rpc.py (which already mocks collect_pending_imports); a new test likely belongs alongside the git integration/component tests exercising collect_pending_imports end-to-end against a real merged branch, or a focused unit test of the new-branches loop behavior.
• Edge case for the fixer/reviewer: confirm whether a merged branch can also surface in the updated_branches loop (repository.py:195) on workers whose local clone still tracks it; in the current code that loop does not call update_commit_value, so the primary fix targets the new-branches loop, but the write-path GraphQLError backstop covers it defensively.

AGENT_ANALYSIS_COMPLETE

Generated by Bug analyst agent for #9931 · 156.8 AIC · ⌖ 29.2 AIC · ⊞ 12.2K · ◷
_{Comment /bug-analyze to run again}

[minitriga]

/bug-tdd

[opsmill-bug-pipeline]

❌ Bug test-writer agent failed. Please review the logs for details.

[polmichel]

/bug-tdd

[polmichel]

/bug-tdd

[opsmill-bug-pipeline]

✅ Bug test-writer agent completed successfully!