Duplicate Prometheus metric definitions generated from datanodes, which causes failure to publish the metrics to grafana

[sokui]

Hi community,

Need some help here. The datanodes in our production generates duplicate prom metrics (duplicate definitions). They seem to be all related to ratis. The following are one of the duplications. This cause the prometheus parsing errors. Do you know what the reason it is, and how to fix it?

# TYPE ratis_leader_election_lastLeaderElapsedTime gauge
ratis_leader_election_lastLeaderElapsedTime{instance="d09c0aaf-dacd-4ad4-95c7-f616ef2906c6",group="group-72DDBFDDFEB2",} 0.0
# TYPE ratis_leader_election_lastLeaderElapsedTime gauge
ratis_leader_election_lastLeaderElapsedTime{instance="d09c0aaf-dacd-4ad4-95c7-f616ef2906c6",group="group-82B743C17C01",} 0.0
# TYPE ratis_leader_election_lastLeaderElapsedTime gauge
ratis_leader_election_lastLeaderElapsedTime{instance="d09c0aaf-dacd-4ad4-95c7-f616ef2906c6",group="group-64B8EF032F77",} 0.0

[kerneltime]

@sokui I believe these metrics are generated from the Datanode and each one corresponds to a Raft ring for a pipeline.
The mental model is as follows: Each Datanode uses multiple raft/ratis rings (3 Datanodes each) to replicate data. Thus each Datanode can host multiple leaders and followers in those raft rings. Each raft ring is identified by a group ID group-72DDBFDDFEB2. Thus, the same metric can be reported multiple times, once for each group.

[kerneltime]

@sokui this should not break Grafana, it is capable of charting the same metric separate by labels. Example: The same metric generated from one node with varying labels

[sokui]

@sokui this should not break Grafana, it is capable of charting the same metric separate by labels. Example: The same metric generated from one node with varying labels

Hi @kerneltime,

Sorry for the confusion. I think the same metrics with different labels are fine. I am talking about the metrics definition, which begins with # Type. I think if we define the same metric multiple times, it will trigger this error line in prometheus code https://github.com/prometheus/common/blob/main/expfmt/text_parse.go#L517

Let me update my title to be more clear.

[adoroszlai]

@sokui Which version of Ozone is this? HDDS-5010 fixed similar problem in 1.2.0. Maybe the fix was incomplete or it got broken since then?

[kerneltime]

@sokui can you try out the local setup outlined here https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/compose/ozone/README.md#add-ons
Feel free to experiment with how ratis collection is setup. Would be interesting to see why this failing for you and not for local setup or the clusters we monitor.

[sokui]

@kerneltime basically, are you saying even with duplicate definitions, your prometheus setup can handle them?

[kerneltime]

@sokui yes and if you follow the steps outlined in the Readme.md you can try it out yourself. That said, we should fix how drop wizard gets setup for multi ratis in Datanode.

[kerneltime]

@sokui did you get to look into this a bit more?

[sokui]

@kerneltime not yet. We will probably do some test next week.