Regression: Application targetting an annotated tag will self heal, even when self-healing is disabled.

[jgwest]

Describe the bug

Note: The cause of this bug is likely the same as #20082. I have opened this current bug to track another failing case, but it likely shares the same root cause as #20082. See that bug for some additional details.

In Git, there are two types of tags: annotated tags and lightweight tags. Annotated tags may have a message/data attached to them, while lightweight tags do not. This means that annotated tags have their own ID (different from the commit SHA), while lightweight tags do not. The ID of the lightweight tag is the same as the target commit SHA.

However, Argo CD behaviour currently differs when .spec.source.targetRevision points to an annotated tag, versus a lightweight tag. This appears to be a regression that was introduced in April 2024, based on 'git bisect' testing.

When:

• you create an Application with automatedSync: true, selfHealing: false
• AND that Application targets an annotated tag,

Then:

• the Application should NOT self heal in any scenario, because selfHealing: false

But at present, selfHealing WILL occur when the targetRevision is an annotated tag, in some cases.

In contrast, when you use a lightweight tag, rather than an annotated tag, the behaviour is correct and as expected (no self healing).

To Reproduce

Automated test:

I wrote an Argo CD E2E test case that reproduces the problem:

• Clone the repo

git clone [email protected]:jgwest/argo-cd
cd argo-cd
git checkout annotated-tag-behaviour-jan-2024
# rebase to main if you wish
make start-e2e

# This will currently fail, as the test is verifying that Argo CD behaviour has regressed: 
go test -count=1 -timeout 30s -run ^TestAutomatedSelfHealingAgainstAnnotatedTag$ github.com/argoproj/argo-cd/v3/test/e2e

# The exact same test, run against a lightweight tag, will pass:
go test -count=1 -timeout 30s -run ^TestAutomatedSelfHealingAgainstLightweightTag$ github.com/argoproj/argo-cd/v3/test/e2e

Manually:

1. For a GitOps repo of your choice, add an annotated tag:

git clone (your gitops repo)
cd "(your gitops repo)"
git tag -f -a my-annotated-tag -m "my annotated tag"
git push -f && git push -f --tag

2. Create Application targetting that annotated tag via .spec.source.targetRevision.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: your-application
  namespace: argocd
spec:
  destination:
    namespace: argocd
    server: https://kubernetes.default.svc
  project: default
  source:
    path: .
    repoURL: # (... your repo URL ...)
    targetRevision: my-annotated-tag
  syncPolicy:
    automated:
      prune: true
      selfHeal: false

3. Wait for it to sync.

4. Update a K8s resource in your gitops repository (for example, updating .spec.revisionHistoryLimit of a Deployment). Push that new commit.

5. Update annotated tag to point to new commit.

• git tag -f -a my-annotated-tag -m "message"
• git push --tags -f origin master

6. Do a hard refresh in the Argo CD UI.

7. Wait for Argo CD to reconcile the K8s resource (e.g. .spec.revisionHistoryLimit of a Deployment) to the new value in Git.

8. Within K8s itself, update that same K8s resource. (e.g. update .spec.revisionHistoryLimit of a Deployment to a value that is different from what is in git).

9. Fail: Even though the Application is selfHeal: false, you will see the change that was made in step 8 is reverted to the previous version of the resource from step 5. (e.g. .spec.revisionHistoryLimit of Deployment has changed back to the value in Git, even though it should not)

Version

See #20082 for detailed description of what caused the regression, but TL;DR: this behaviour is currently present in v2.11.x, v2.12.x, v2.13.x and current master.

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.