Feature Flags are loaded randomly in the ConfigMap

[simonmalletrc]

Hello,

We have just finished configuring AppConfiguration Kubernetes Provider in our infrastructure and everything seems to be working except for the feature flags.

When I use the spec.featureFlag.refresh in conjuction with stakater/reloader, every refresh it does a rolling update.

Rolling updates are done only when changes are detected.
After investigation we found out that the order in which feature flags are loaded in the ConfigMap are completely random instead of being in alphabetical order which triggers a rolling update every refresh.

        featureFlag: #Optional. it is the content spec.featureFlag in the doc https://learn.microsoft.com/en-ca/azure/azure-app-configuration/reference-kubernetes-provider?tabs=default.
          selectors:
          - keyFilter: "*"
          - keyFilter: "*"
            labelFilter: dev
          - keyFilter: "*"
            labelFilter: dev35
          refresh:
            enabled: true
            interval: 1m

Expected Result

Rolling update only when actual changes

Actual Result

Rolling update with same Feature Flag values (because of feature flags being loaded randomly in the ConfigMap)

[linglingye001]

Hi @simonmalletrc , currently the Kubernetes provider pulls the latest feature flag settings from the app config store every refresh interval and regenerate the ConfigMap (even if the content hasn't changed). We already have PR aimed at optimizing this refresh process, so that the ConfigMap will only be updated when the selected keyValues/featureFlags really change. This enhancement will be included in the next version, please stay tuned.

[simonmalletrc]

Thank you !

[RichardChen820]

It's been fixed in the latest v2.1.0.

[simonmalletrc]

I confirm it works as intended in version 2.1.0
Thank you!

[simonmalletrc]

@RichardChen820
I must reopen this ticket because part of the implementation is not working as intended.

What is working

When a refresh occurs, the configMap is not updated if feature manager values have not changed. Which means the container is not restarted for no reason. Great!

What is not working

Based on the documentation here:
How to trigger on-demand refresh of ConfigMap and Secret
https://learn.microsoft.com/en-us/azure/azure-app-configuration/quickstart-azure-kubernetes-service#how-to-trigger-on-demand-refresh-of-configmap-and-secret

When I restart the controller using the command
kubectl rollout restart deployment az-appconfig-k8s-provider -n azappconfig-system

it refreshes the configmap and the order of the keys in the feature manager changes which triggers a rolling update although nothing has changed.

Expected behaviour

The keys inside the feature_management object should be in alphabetical order to avoid changing position and triggering a rolling update. It is working as expected for values inside the configuration explorer. Only the Feature Manager is affected.

[simonmalletrc]

values.yaml for reference

AzureAppConfigurationProvider: #IMP: the key workload-identity should be set and the identity should have permission on the azure rsource "App Configuration". this is a custom resource the cluster should have installed the extension "Microsoft.AppConfiguration"
      appconfigurationprovider: #name of the custom resource.
        configuration:
          selectors:
            - keyFilter: "scoop.astro.api*"
            - keyFilter: "scoop.astro.api*"
              labelFilter: "dev"
            - keyFilter: "scoop.astro.api*"
              labelFilter: "dev35"
          refresh:
            enabled: false
            interval: 10m
            monitoring:
              keyValues:
                - key: scoop.astro.api.sentinel
                  label: dev35
        endpoint: https://store_name_xyz.azconfig.io #this is the endpoint of the azure resource "App Configuration"
        featureFlag: #Optional. it is the content spec.featureFlag in the doc https://learn.microsoft.com/en-ca/azure/azure-app-configuration/reference-kubernetes-provider?tabs=default.
          selectors:
          - keyFilter: "*"
          - keyFilter: "*"
            labelFilter: dev
          - keyFilter: "*"
            labelFilter: dev35
          refresh:
            enabled: false
            interval: 10m
        configMap:
          name: configmap # the content of the azure resource "App Configuration" will be injected in the cluster as configmap. this is the name of the configmap
          configMapData:
            key: configmap.json #this is the key in the configmap
            type: json
          mountPoint:
            - containerName: node
              volMountPoint: /etc/configmap
        keyVaultSecret:
          name: keyvaultsecret
          mountPoint: #OPTIONAL. required if keyVaultSecretName is set
            - containerName: node
              volMountPoint: /etc/keyvault

[RichardChen820]

@simonmalletrc Thanks for reporting this, this is indeed an issue, we need to ensure the configMap is refreshed only when there are actual content changes. The current problem lies in the serialization process, as converting objects into string value does not produce consistent results, we will work on fixing it. For now, you might could avoid frequently restarting the controller unless absolutely necessary.

[RichardChen820]

@linglingye001 The potential fix may require covering these two points:

1. Ensure the serialization process can produce consistent results
2. Using a hash algorithm to generate a digest for comparing whether the content has changed.