Application sync loops leaving the App in OutOfSync status #21497
Comments
nitishfy
added
component:application-controller
version:2.13
|
I think to help understand the issue, you should add screenshots of your Apps of apps structure, as well as Application/AppSet example manifests so it is easier to visualize and reproduce. From what I understand from the description, the flow would be: How do you determine that "the sync finishes successfully"? What is the status of your resource at that moment? |
|
@agaudreault here are the example manifests with the exact spec we deploy, all appset we deploy have this spec, only names and paths for sources change. It did not allow me to add it to the first message because of length limitations ApplicationSet generated Applications example screenshot
App of apps example manifestapiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: services
namespace: argocd
spec:
project: default
sources:
- repoURL: https://github.com/example/argocd-applications.git
targetRevision: main
path: .environments/staging # kustomization that generates the appsets manifests
destination:
server: https://kubernetes.default.svc
namespace: argocd
revisionHistoryLimit: 5
syncPolicy:
automated:
prune: trueApplicationSet example manifestapiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
annotations:
baseChartVersion: '>=2.0.0'
labels:
argocd.argoproj.io/instance: services
name: example-svc
namespace: argocd
spec:
generators:
- list:
elements:
- infra: true
type: infra
- infra: false
type: app
goTemplate: true
strategy:
rollingSync:
steps:
- matchExpressions:
- key: appType
operator: In
values:
- infra
- matchExpressions:
- key: appType
operator: In
values:
- app
type: RollingSync
template:
metadata:
annotations:
argocd.argoproj.io/manifest-generate-paths: /example-svc/staging
labels:
appType: '{{.type}}'
name: example-svc-{{.type}}
namespace: argocd
spec:
destination:
namespace: services
server: https://kubernetes.default.svc
ignoreDifferences:
- jsonPointers:
- /data/password
kind: Secret
name: accounting-svc-db-creds
- group: iam.aws.crossplane.io
jqPathExpressions:
- .spec.forProvider.tags[] | select(.key | startswith("crossplane"))
kind: Role
project: services
revisionHistoryLimit: 5
sources:
- chart: '{{.type}}-chart'
helm:
ignoreMissingValueFiles: true
parameters:
- name: global.environment
value: staging
- name: global.owner
value: team-example
releaseName: example-svc
valueFiles:
- $values/example-svc/staging/values.yaml
- $values/example-svc/staging/secrets.yaml
repoURL: 000000000000.dkr.ecr.eu-west-1.amazonaws.com
targetRevision: '>=2.0.0'
- ref: values
repoURL: https://github.com/example/argocd-applications
targetRevision: main
syncPolicy:
automated:
allowEmpty: true
prune: true
selfHeal: true
retry:
backoff:
duration: 5m
factor: 1
maxDuration: 5m
limit: -1
syncOptions:
- ApplyOutOfSyncOnly=true
- CreateNamespace=false
- FailOnSharedResource=true
- PruneLast=true |
|
The flow you describe is exactly what happens. Then, after the last step, it returns to the second step you described, and it loops endlessly. I determine that "the sync finishes successfully" when the diff of all Sync stages and waves have been applied and, therefore, the application is 100% synced (App status is |
Context
We have an app of apps, with auto-sync enabled, managing several appsets, which represent a service each. These appsets are composed of two applications that are rolled out using rollingSync. The former application deploys infra resources (DBs, buckets, etc with crossplane). The latter deploys the service per se (deployments, services, cronjobs, configmaps, etc), and has a pre-sync hook to run database migrations. Both applications in the appsets are composed of multiple sources, an OCI helm chart plus sourcing values.yaml files stored in a GitHub repository.
Describe the bug
Sometimes (I could not find anything in common yet), when the latter app sources have been updated and the app sync gets triggered, the pre-sync hook runs successfully, but at this point the sync finishes "successfully" without applying any of the resources in the main Sync stage. Therefore, it finishes the sync leaving the app in an outofsync state. So, the app sync gets triggered again, with the same result. The loop continues endlessly.
The workaround to get the app synced correctly is non-cascade deleting the buggy app. Then, the appset creates the app again and the sync runs successfully this time.
Custom configuration
argo-cd helm chart custom values
Screenshots
ArgoCD notifications screenshot
Version
Logs
ArgoCD application controller logs
The text was updated successfully, but these errors were encountered: