Skip to content

[pod logs] Use the K8s deployment/cronjob/job name in the service_name logic #1540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
cyrille-leclerc opened this issue May 13, 2025 · 2 comments · May be fixed by #1549
Open

[pod logs] Use the K8s deployment/cronjob/job name in the service_name logic #1540

cyrille-leclerc opened this issue May 13, 2025 · 2 comments · May be fixed by #1549
Assignees
@petewall

Comments

@cyrille-leclerc
Copy link
Collaborator

cyrille-leclerc commented May 13, 2025 •
edited
Loading

We want to align the service_name logic of the K8s Monitoring Helm Chart Pod Logs feature on the logic used by the OpenTelemetry Operator.

For this, we need to retrieve the deployment/cronjob/job of the pods through the "controller-by" attribute.
If the OpenTelemetry Collector K8s Processor is capable of retrieving these metadata, it's not clear if Alloy's discovery.relabel can also do it.

OpenTelemetry Operator logic for service.name

First value found:

  • pod.annotation[resource.opentelemetry.io/service.name]
  • if (config[useLabelsForResourceAttributes]) pod.label[app.kubernetes.io/name]
  • k8s.deployment.name
  • k8s.replicaset.name
  • k8s.statefulset.name
  • k8s.daemonset.name
  • k8s.cronjob.name
  • k8s.job.name
  • k8s.pod.name
  • k8s.container.name

See also
@cyrille-leclerc cyrille-leclerc added the enhancement New feature or request label May 13, 2025
@cyrille-leclerc
Copy link
Collaborator Author

There may be a hack looking at __meta_kubernetes_pod_controller_kind (e.g. ReplicaSet) and __meta_kubernetes_pod_controller_name (e.g. orders-prometheus-postgres-exporter-6f65896bd5) and extracting the substringBeforeLast(__meta_kubernetes_pod_controller_name, '-').

{
	__address__                                              = "10.0.12.114:9187",
	__meta_kubernetes_namespace                              = "ditl-demo-prod",
	__meta_kubernetes_pod_annotation_checksum_config         = "01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b",
	__meta_kubernetes_pod_annotation_checksum_secrets        = "4d8241978bef61f991832cebf24c44d8aad147d9a9629663a01a24c05ec414d7",
	__meta_kubernetes_pod_annotationpresent_checksum_config  = "true",
	__meta_kubernetes_pod_annotationpresent_checksum_secrets = "true",
	__meta_kubernetes_pod_container_id                       = "containerd://bbb03a33212e7ce2b5a649c5a460df4e6ad6a460c959f7d1da811eee226b08b7",
	__meta_kubernetes_pod_container_image                    = "quay.io/prometheuscommunity/postgres-exporter:v0.17.1",
	__meta_kubernetes_pod_container_init                     = "false",
	__meta_kubernetes_pod_container_name                     = "prometheus-postgres-exporter",
	__meta_kubernetes_pod_container_port_name                = "http",
	__meta_kubernetes_pod_container_port_number              = "9187",
	__meta_kubernetes_pod_container_port_protocol            = "TCP",
	__meta_kubernetes_pod_controller_kind                    = "ReplicaSet",
	__meta_kubernetes_pod_controller_name                    = "orders-prometheus-postgres-exporter-6f65896bd5",
	__meta_kubernetes_pod_host_ip                            = "10.0.12.211",
	__meta_kubernetes_pod_ip                                 = "10.0.12.114",
	__meta_kubernetes_pod_label_app                          = "prometheus-postgres-exporter",
	__meta_kubernetes_pod_label_pod_template_hash            = "6f65896bd5",
	__meta_kubernetes_pod_label_release                      = "orders",
	__meta_kubernetes_pod_labelpresent_app                   = "true",
	__meta_kubernetes_pod_labelpresent_pod_template_hash     = "true",
	__meta_kubernetes_pod_labelpresent_release               = "true",
	__meta_kubernetes_pod_name                               = "orders-prometheus-postgres-exporter-6f65896bd5-rwfct",
	__meta_kubernetes_pod_node_name                          = "ip-10-0-12-211.us-west-2.compute.internal",
	__meta_kubernetes_pod_phase                              = "Running",
	__meta_kubernetes_pod_ready                              = "true",
	__meta_kubernetes_pod_uid                                = "d52e8acb-971c-48f6-badf-726e9000bea4",
}, {
	__address__                                              = "10.0.12.176:9187",
	__meta_kubernetes_namespace                              = "ditl-demo-prod",
	__meta_kubernetes_pod_annotation_checksum_config         = "01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b",
	__meta_kubernetes_pod_annotation_checksum_secrets        = "8daa3ce1a12d2d9ba4af63b10f7cebe378bcd68908bdb369e0726c2d1ef5d5f1",
	__meta_kubernetes_pod_annotationpresent_checksum_config  = "true",
	__meta_kubernetes_pod_annotationpresent_checksum_secrets = "true",
	__meta_kubernetes_pod_container_id                       = "containerd://f0bf7a8288f5c7bbf3e5b77334b1153e32d59ef5fd1586bb8c78ff4febb7955b",
	__meta_kubernetes_pod_container_image                    = "quay.io/prometheuscommunity/postgres-exporter:v0.17.1",
	__meta_kubernetes_pod_container_init                     = "false",
	__meta_kubernetes_pod_container_name                     = "prometheus-postgres-exporter",
	__meta_kubernetes_pod_container_port_name                = "http",
	__meta_kubernetes_pod_container_port_number              = "9187",
	__meta_kubernetes_pod_container_port_protocol            = "TCP",
	__meta_kubernetes_pod_controller_kind                    = "ReplicaSet",
	__meta_kubernetes_pod_controller_name                    = "productcatalog-prometheus-postgres-exporter-6768c4cd7c",
	__meta_kubernetes_pod_host_ip                            = "10.0.12.211",
	__meta_kubernetes_pod_ip                                 = "10.0.12.176",
	__meta_kubernetes_pod_label_app                          = "prometheus-postgres-exporter",
	__meta_kubernetes_pod_label_pod_template_hash            = "6768c4cd7c",
	__meta_kubernetes_pod_label_release                      = "productcatalog",
	__meta_kubernetes_pod_labelpresent_app                   = "true",
	__meta_kubernetes_pod_labelpresent_pod_template_hash     = "true",
	__meta_kubernetes_pod_labelpresent_release               = "true",
	__meta_kubernetes_pod_name                               = "productcatalog-prometheus-postgres-exporter-6768c4cd7c-wjmxl",
	__meta_kubernetes_pod_node_name                          = "ip-10-0-12-211.us-west-2.compute.internal",
	__meta_kubernetes_pod_phase                              = "Running",
	__meta_kubernetes_pod_ready                              = "true",
	__meta_kubernetes_pod_uid                                = "7ffa9251-eb9f-4e04-848a-962c038f3999",
}, {
	__address__                                                    = "10.0.12.224:53",
	__meta_kubernetes_namespace                                    = "kube-system",
	__meta_kubernetes_pod_container_id                             = "containerd://34126b768b988df8289ff09e4849bd22da373d04eeaa7b7f8b28d4163e7136ae",
	__meta_kubernetes_pod_container_image                          = "602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/coredns:v1.11.4-eksbuild.10",
	__meta_kubernetes_pod_container_init                           = "false",
	__meta_kubernetes_pod_container_name                           = "coredns",
	__meta_kubernetes_pod_container_port_name                      = "dns",
	__meta_kubernetes_pod_container_port_number                    = "53",
	__meta_kubernetes_pod_container_port_protocol                  = "UDP",
	__meta_kubernetes_pod_controller_kind                          = "ReplicaSet",
	__meta_kubernetes_pod_controller_name                          = "coredns-5c8b6956d4",
	__meta_kubernetes_pod_host_ip                                  = "10.0.12.211",
	__meta_kubernetes_pod_ip                                       = "10.0.12.224",
	__meta_kubernetes_pod_label_eks_amazonaws_com_component        = "coredns",
	__meta_kubernetes_pod_label_k8s_app                            = "kube-dns",
	__meta_kubernetes_pod_label_pod_template_hash                  = "5c8b6956d4",
	__meta_kubernetes_pod_labelpresent_eks_amazonaws_com_component = "true",
	__meta_kubernetes_pod_labelpresent_k8s_app                     = "true",
	__meta_kubernetes_pod_labelpresent_pod_template_hash           = "true",
	__meta_kubernetes_pod_name                                     = "coredns-5c8b6956d4-5vvhv",
	__meta_kubernetes_pod_node_name                                = "ip-10-0-12-211.us-west-2.compute.internal",
	__meta_kubernetes_pod_phase                                    = "Running",
	__meta_kubernetes_pod_ready                                    = "true",
	__meta_kubernetes_pod_uid                                      = "597b98bb-bb78-4def-8d4b-8cc776e3abe7",
}, {
	__address__                                                    = "10.0.12.224:53",
	__meta_kubernetes_namespace                                    = "kube-system",
	__meta_kubernetes_pod_container_id                             = "containerd://34126b768b988df8289ff09e4849bd22da373d04eeaa7b7f8b28d4163e7136ae",
	__meta_kubernetes_pod_container_image                          = "602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/coredns:v1.11.4-eksbuild.10",
	__meta_kubernetes_pod_container_init                           = "false",
	__meta_kubernetes_pod_container_name                           = "coredns",
	__meta_kubernetes_pod_container_port_name                      = "dns-tcp",
	__meta_kubernetes_pod_container_port_number                    = "53",
	__meta_kubernetes_pod_container_port_protocol                  = "TCP",
	__meta_kubernetes_pod_controller_kind                          = "ReplicaSet",
	__meta_kubernetes_pod_controller_name                          = "coredns-5c8b6956d4",
	__meta_kubernetes_pod_host_ip                                  = "10.0.12.211",
	__meta_kubernetes_pod_ip                                       = "10.0.12.224",
	__meta_kubernetes_pod_label_eks_amazonaws_com_component        = "coredns",
	__meta_kubernetes_pod_label_k8s_app                            = "kube-dns",
	__meta_kubernetes_pod_label_pod_template_hash                  = "5c8b6956d4",
	__meta_kubernetes_pod_labelpresent_eks_amazonaws_com_component = "true",
	__meta_kubernetes_pod_labelpresent_k8s_app                     = "true",
	__meta_kubernetes_pod_labelpresent_pod_template_hash           = "true",
	__meta_kubernetes_pod_name                                     = "coredns-5c8b6956d4-5vvhv",
	__meta_kubernetes_pod_node_name                                = "ip-10-0-12-211.us-west-2.compute.internal",
	__meta_kubernetes_pod_phase                                    = "Running",
	__meta_kubernetes_pod_ready                                    = "true",
	__meta_kubernetes_pod_uid                                      = "597b98bb-bb78-4def-8d4b-8cc776e3abe7",
}

@petewall
Copy link
Collaborator

We could do this if controller is DaemonSet or StatefulSet. If the controller is "ReplicaSet" or "Job", we can't really make the assumption that it's from a RS or Deployment, or from a Job or CronJob. It's too ambiguous...

Perhaps a better, long term idea would be to use something like this:
https://grafana.com/docs/alloy/latest/reference/components/loki/loki.enrich

@petewall petewall self-assigned this May 14, 2025
@petewall petewall removed the enhancement New feature or request label May 14, 2025
@petewall petewall linked a pull request May 14, 2025 that will close this issue
Set service_name from k8s owner #1549
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@petewall petewall

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Set service_name from k8s owner grafana/k8s-monitoring-helm
2 participants
@cyrille-leclerc @petewall