Ability to define different annotations for annotation-based auto-discovery

[snk-actian]

We were using Grafana Agent static mode and then upgraded to Flow mode. We are using Grafana Agent Flow mode in all our environments for approx a year.

Recently we upgraded our development env to k8s-monitoring-helm v1 and planning to upgrade to v2. Then we will move forward to higher envs for sure.

Our problem with v1 (and as I see same on v2) is as follows:

Historically most of the applications running on our clusters were annotated with prometheus scheme, for example

prometheus.io/scrape
prometheus.io/port

Then when we upgraded to Flow mode, we changed those annotations with Grafana Agent Flow mode scheme, for example:

metrics.agent.grafana.com/scrape
metrics.agent.grafana.com/port

Same goes for log related annotations.

Now with k8s-monitoring-helm v1, I have defined the following to keep our annotations (not to ask engineers to change once more 😬 ):

metrics:
  autoDiscover:
    annotations:
      scrape: "metrics.agent.grafana.com/scrape"
      port: "metrics.agent.grafana.com/port"
      ...
      ...
      ... all other possible annotations to replace with Flowmode scheme
      ...
      ...

I see, there are ways to define similar in v2 as well...

Recently I have realized that there are some community applications / Helm charts for enterprise applications we use which does not provide a way to define those annotations easily. Yes this is (kind of) bad Helm Chart but this is what we have today

Question: Can there be a way to define multiple options for those annotations. So we can use both prometheus.io scheme and Flow Mode scheme, like a regular expression format prometheus.io/scrape|metrics,agent.grafana.com/scrape (or any other way)??

I am planning to move slowly to new k8s-monitoring-helm scheme as in k8s.grafana.com/... but without forcing engineering or without breaking anything. So this will help for such transitions and for such Helm charts which do not allow annotations override

[petewall]

I think the best way to solve this might be to introduce a "preFilterDiscoveryRule" section that would let you do something like:

rule {
  source_labels = ["__meta_kubernetes_pod_label_prometheus_io_scrape"]
  target_label = "__meta_kubernetes_pod_label_k8s_grafana_com_scrape"
}
rule {
  source_labels = ["__meta_kubernetes_pod_label_prometheus_io_port"]
  target_label = "__meta_kubernetes_pod_label_k8s_grafana_com_port"
}

This would, in Alloy, rewrite the prometheus.io annotations into the k8s.grafana.com annotations and then get picked up by the rest of the path.