From d060fc92d37aaa06122aa234e8f9ae8f08caff9c Mon Sep 17 00:00:00 2001 From: Taylor Cole Date: Wed, 18 Sep 2024 12:53:17 -0700 Subject: [PATCH 01/16] docs: adds instructions for collecting and sending native histograms with otel collector --- .../mimir/send/native-histograms/_index.md | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/docs/sources/mimir/send/native-histograms/_index.md b/docs/sources/mimir/send/native-histograms/_index.md index 79a40811958..67a9b045049 100644 --- a/docs/sources/mimir/send/native-histograms/_index.md +++ b/docs/sources/mimir/send/native-histograms/_index.md @@ -184,6 +184,33 @@ Use the latest version of [Grafana Alloy](https://grafana.com/docs/alloy/ Date: Thu, 19 Sep 2024 09:15:13 -0700 Subject: [PATCH 02/16] Update docs/sources/mimir/send/native-histograms/_index.md Co-authored-by: George Krajcsovits --- docs/sources/mimir/send/native-histograms/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/mimir/send/native-histograms/_index.md b/docs/sources/mimir/send/native-histograms/_index.md index 67a9b045049..5876464d801 100644 --- a/docs/sources/mimir/send/native-histograms/_index.md +++ b/docs/sources/mimir/send/native-histograms/_index.md @@ -184,7 +184,7 @@ Use the latest version of [Grafana Alloy](https://grafana.com/docs/alloy/ Date: Thu, 19 Sep 2024 09:15:41 -0700 Subject: [PATCH 03/16] Update docs/sources/mimir/send/native-histograms/_index.md Co-authored-by: George Krajcsovits --- docs/sources/mimir/send/native-histograms/_index.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/sources/mimir/send/native-histograms/_index.md b/docs/sources/mimir/send/native-histograms/_index.md index 5876464d801..4bd181035c0 100644 --- a/docs/sources/mimir/send/native-histograms/_index.md +++ b/docs/sources/mimir/send/native-histograms/_index.md @@ -186,7 +186,15 @@ Use the latest version of [Grafana Alloy](https://grafana.com/docs/alloy/ Date: Thu, 19 Sep 2024 09:15:58 -0700 Subject: [PATCH 04/16] Update docs/sources/mimir/send/native-histograms/_index.md Co-authored-by: George Krajcsovits --- docs/sources/mimir/send/native-histograms/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/mimir/send/native-histograms/_index.md b/docs/sources/mimir/send/native-histograms/_index.md index 4bd181035c0..999f759f1ca 100644 --- a/docs/sources/mimir/send/native-histograms/_index.md +++ b/docs/sources/mimir/send/native-histograms/_index.md @@ -198,7 +198,7 @@ OpenTelemetry supports instrumenting applications with the OpenTelemetry SDK. Mu ### Instrument the OpenTelemetry Collector using Go -Use the OpenTelemetry SDK version 1.17.0 or later. +Use the OpenTelemetry SDK version 1.17.0 or later for Go. 1. Set up the collector to handle your metrics data. This includes setting up your resources, meter provider, meter, instruments, and views. Refer to [Metrics](https://opentelemetry.io/docs/languages/go/instrumentation/#metrics) in the OpenTelemetry SDK documentation for Go. 1. To aggregate a histogram instrument as an exponential histogram, include the following view: From b3a8d7fad9356973595bc2e0893a5b457207fe58 Mon Sep 17 00:00:00 2001 From: Taylor C <41653732+tacole02@users.noreply.github.com> Date: Thu, 19 Sep 2024 09:16:52 -0700 Subject: [PATCH 05/16] Update docs/sources/mimir/send/native-histograms/_index.md Co-authored-by: George Krajcsovits --- docs/sources/mimir/send/native-histograms/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/mimir/send/native-histograms/_index.md b/docs/sources/mimir/send/native-histograms/_index.md index 999f759f1ca..ca1c3d169bb 100644 --- a/docs/sources/mimir/send/native-histograms/_index.md +++ b/docs/sources/mimir/send/native-histograms/_index.md @@ -196,7 +196,7 @@ Grafana Mimir can receive exponential histograms as is via the OTLP protocol or OpenTelemetry supports instrumenting applications with the OpenTelemetry SDK. Multiple languages are supported, for the complete list see [Language APIs & SDKs](https://opentelemetry.io/docs/languages/). -### Instrument the OpenTelemetry Collector using Go +### Instrument a Go application with the OpenTelemetry SDK Use the OpenTelemetry SDK version 1.17.0 or later for Go. From cb17a5eb30baa053250d3a870c644f54c428b208 Mon Sep 17 00:00:00 2001 From: Taylor Cole Date: Thu, 19 Sep 2024 12:14:00 -0700 Subject: [PATCH 06/16] docs: add new topic for exponential histograms --- .../mimir/send/native-histograms/_index.md | 35 ------------------ .../otel-exponential-histograms/_index.md | 36 +++++++++++++++++++ 2 files changed, 36 insertions(+), 35 deletions(-) create mode 100644 docs/sources/mimir/send/otel-exponential-histograms/_index.md diff --git a/docs/sources/mimir/send/native-histograms/_index.md b/docs/sources/mimir/send/native-histograms/_index.md index ca1c3d169bb..79a40811958 100644 --- a/docs/sources/mimir/send/native-histograms/_index.md +++ b/docs/sources/mimir/send/native-histograms/_index.md @@ -184,41 +184,6 @@ Use the latest version of [Grafana Alloy](https://grafana.com/docs/alloy/ Date: Thu, 19 Sep 2024 15:51:32 -0700 Subject: [PATCH 07/16] docs: otel histogram details --- .../otel-exponential-histograms/_index.md | 31 ++++++++++++------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/docs/sources/mimir/send/otel-exponential-histograms/_index.md b/docs/sources/mimir/send/otel-exponential-histograms/_index.md index 2856050b366..cdf14c0f6c1 100644 --- a/docs/sources/mimir/send/otel-exponential-histograms/_index.md +++ b/docs/sources/mimir/send/otel-exponential-histograms/_index.md @@ -1,27 +1,34 @@ --- description: Learn how to collect and send exponential histograms with the OpenTelemetry Collector +keywords: + - send metrics + - exponential histogram + - OpenTelemetry + - instrumentation menuTitle: OpenTelemetry exponential histograms -title: Collect and send exponential histograms with the OpenTelemetry Collector -weight: 100 +title: Send exponential histograms to Mimir +weight: 200 --- -# Collect and send exponential histograms with the OpenTelemetry Collector +# Send exponential histograms to Mimir -OpenTelemetry [exponential histograms](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponentialhistogram) are compatible with Prometheus native histograms. The only significant difference is that exponential histograms store the `min` and `max` observation values explicitly, but native histograms don't, which means you need to estimate the minimum and maximum by using the 0.0 and 1.0 quantile respectively. +You can collect and send exponential histograms to Mimir with the OpenTelemetry Collector. OpenTelemetry [exponential histograms](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponentialhistogram) are compatible with Prometheus native histograms. The key difference is that exponential histograms store the `min` and `max` observation values explicitly, whereas native histograms don't. This means that for exponential histograms, you don't need to estimate these values using the 0.0 and 1.0 quantiles. -Use the OpenTelemetry Collector to collect and send exponential histograms. The Collector supports collecting exponential histograms or other compatible data formats (for example native histograms, DataDog sketches, etc) through its receivers. The Collector also supports sending the exponential histograms or compatible data formats such as native histograms through its exporters. +The OpenTelemetry Collector supports collecting exponential histograms and other compatible data formats, such as native histograms and DataDog sketches, through its receivers and sending them through its exporters. -Note: the availability of different receivers and exporters depends on your Collector [distribution](https://opentelemetry.io/docs/concepts/distributions/). +{{< admonition type="note" >}} +The availability of different receivers and exporters depends on your Collector [distribution](https://opentelemetry.io/docs/concepts/distributions/). +{{< /admonition >}} -Grafana Mimir can receive exponential histograms as is via the OTLP protocol or as Prometheus native histograms over the Prometheus remote write protocol. +You can use the OpenTelemetry (OTLP) protocol to send exponential histograms to Grafana Mimir in their existing format, or you can use the Prometheus remote write protocol to send them as Prometheus native histograms. -OpenTelemetry supports instrumenting applications with the OpenTelemetry SDK. Multiple languages are supported, for the complete list see [Language APIs & SDKs](https://opentelemetry.io/docs/languages/). +The OpenTelemetry SDK supports instrumenting applications in multiple languages. Refer to [Language APIs & SDKs](https://opentelemetry.io/docs/languages/). -## Instrument a Go application with the OpenTelemetry SDK +## Instrument an application with the OpenTelemetry SDK using Go -Use the OpenTelemetry SDK version 1.17.0 or later for Go. +Use the OpenTelemetry SDK version 1.17.0 or later. -1. Set up the collector to handle your metrics data. This includes setting up your resources, meter provider, meter, instruments, and views. Refer to [Metrics](https://opentelemetry.io/docs/languages/go/instrumentation/#metrics) in the OpenTelemetry SDK documentation for Go. +1. Set up the OpenTelemetry Collector to handle your metrics data. This includes setting up your resources, meter provider, meter, instruments, and views. Refer to [Metrics](https://opentelemetry.io/docs/languages/go/instrumentation/#metrics) in the OpenTelemetry SDK documentation for Go. 1. To aggregate a histogram instrument as an exponential histogram, include the following view: ``` @@ -34,3 +41,5 @@ Use the OpenTelemetry SDK version 1.17.0 or later for Go. For more information, refer to [Registering Views](https://opentelemetry.io/docs/languages/go/instrumentation/#registering-views) in the OpenTelemetry SDK documentation for Go. ## Migrate from classic histograms + +## Bucket boundary calculation \ No newline at end of file From 3ba7ad2a3c49ae49c4c209ea15e16d959a6dcfff Mon Sep 17 00:00:00 2001 From: Taylor Cole Date: Thu, 19 Sep 2024 16:04:46 -0700 Subject: [PATCH 08/16] docs: otel histogram details --- .../mimir/send/otel-exponential-histograms/_index.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/sources/mimir/send/otel-exponential-histograms/_index.md b/docs/sources/mimir/send/otel-exponential-histograms/_index.md index cdf14c0f6c1..318a2e353fb 100644 --- a/docs/sources/mimir/send/otel-exponential-histograms/_index.md +++ b/docs/sources/mimir/send/otel-exponential-histograms/_index.md @@ -6,21 +6,21 @@ keywords: - OpenTelemetry - instrumentation menuTitle: OpenTelemetry exponential histograms -title: Send exponential histograms to Mimir +title: Send OpenTelemetry exponential histograms to Mimir weight: 200 --- -# Send exponential histograms to Mimir +# Send OpenTelemetry exponential histograms to Mimir You can collect and send exponential histograms to Mimir with the OpenTelemetry Collector. OpenTelemetry [exponential histograms](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponentialhistogram) are compatible with Prometheus native histograms. The key difference is that exponential histograms store the `min` and `max` observation values explicitly, whereas native histograms don't. This means that for exponential histograms, you don't need to estimate these values using the 0.0 and 1.0 quantiles. The OpenTelemetry Collector supports collecting exponential histograms and other compatible data formats, such as native histograms and DataDog sketches, through its receivers and sending them through its exporters. {{< admonition type="note" >}} -The availability of different receivers and exporters depends on your Collector [distribution](https://opentelemetry.io/docs/concepts/distributions/). +The availability of different receivers and exporters depends on your OpenTelemetry Collector [distribution](https://opentelemetry.io/docs/concepts/distributions/). {{< /admonition >}} -You can use the OpenTelemetry (OTLP) protocol to send exponential histograms to Grafana Mimir in their existing format, or you can use the Prometheus remote write protocol to send them as Prometheus native histograms. +You can use the OpenTelemetry (OTLP) protocol to send exponential histograms to Grafana Mimir in their existing format, or you can use the Prometheus remote write protocol to send them as Prometheus native histograms. Refer to [Send native histograms to Mimir](https://grafana.com/docs/mimir//send/native-histograms/). The OpenTelemetry SDK supports instrumenting applications in multiple languages. Refer to [Language APIs & SDKs](https://opentelemetry.io/docs/languages/). From 2ad64e568bf08c2c538556ea55d142dcef2536d7 Mon Sep 17 00:00:00 2001 From: Taylor Cole Date: Mon, 23 Sep 2024 17:01:10 -0700 Subject: [PATCH 09/16] docs: add migration instructions for exp histograms --- .../mimir/send/native-histograms/_index.md | 2 +- .../otel-exponential-histograms/_index.md | 73 ++++++++++++++++++- 2 files changed, 72 insertions(+), 3 deletions(-) diff --git a/docs/sources/mimir/send/native-histograms/_index.md b/docs/sources/mimir/send/native-histograms/_index.md index 79a40811958..12543d4f442 100644 --- a/docs/sources/mimir/send/native-histograms/_index.md +++ b/docs/sources/mimir/send/native-histograms/_index.md @@ -224,7 +224,7 @@ To ease the migration process, you can keep the custom bucket definition of a cl 1. Send native histograms to remote write, along with the existing classic histograms. 1. Modify dashboards to use the native histogram metrics. Refer to [Visualize native histograms](https://grafana.com/docs/mimir//visualize/native-histograms/) for more information. - Use one of the following strategies to update dashbaords. + Use one of the following strategies to update dashboards. - (Recommended) Add new dashboards with the new native histogram queries. This solution requires looking at different dashboards for data before and after the migration, until data before the migration is removed due to passing its retention time. You can publish the new dashboard when sufficient time has passed to serve users with the new data. - Add a dashboard variable to your dashboard to enable switching between classic histograms and native histograms. There isn't support for selectively enabling and disabling queries in Grafana ([issue 79848](https://github.com/grafana/grafana/issues/79848)). As a workaround, add the dashboard variable `latency_metrics`, for example, and assign it a value of either `-1` or `1`. Then, add the following two queries to the panel: diff --git a/docs/sources/mimir/send/otel-exponential-histograms/_index.md b/docs/sources/mimir/send/otel-exponential-histograms/_index.md index 318a2e353fb..702221e5427 100644 --- a/docs/sources/mimir/send/otel-exponential-histograms/_index.md +++ b/docs/sources/mimir/send/otel-exponential-histograms/_index.md @@ -38,8 +38,77 @@ Use the OpenTelemetry SDK version 1.17.0 or later. } ``` - For more information, refer to [Registering Views](https://opentelemetry.io/docs/languages/go/instrumentation/#registering-views) in the OpenTelemetry SDK documentation for Go. + For more information about views, refer to [Registering Views](https://opentelemetry.io/docs/languages/go/instrumentation/#registering-views) in the OpenTelemetry SDK documentation for Go. For information about view configuration parameters, refer to [Base2 Exponential Bucket Histogram Aggregation](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk.md#base2-exponential-bucket-histogram-aggregation) in the OpenTelemetry Metrics SDK on GitHub. -## Migrate from classic histograms +## Migrate from explicit bucket histograms + +To ease the migration process, you can keep the custom bucket definition of an explicit bucket histogram and add a view for the exponential histogram. + +1. Start with an existing histogram that uses explicit buckets. +1. Create a view for the exponential histogram. Assign this view a unique name and include the exponential aggregation. This creates a metric with the assigned name and exponential buckets. + + The following example shows how to create a metric called `request_latency_exp` that uses exponential buckets. + + ``` + v := sdkmetric.NewView(sdkmetric.Instrument{ + Name: "request_latency", + Kind: sdkmetric.InstrumentKindHistogram, + }, sdkmetric.Stream{ + Name: "request_latency_exp", + Aggregation: sdkmetric.AggregationBase2ExponentialHistogram{MaxSize: 160, NoMinMax: true, MaxScale: 20}, + //Aggregation: sdkmetric.DefaultAggregationSelector(sdkmetric.InstrumentKindHistogram), + }) + ``` + + For more information about creating a view, refer to [Views](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#view) in the OpenTelemetry Metrics SDK. +1. Modify dashboards to use the exponential histogram metrics. Refer to [Visualize native histograms](https://grafana.com/docs/mimir//visualize/native-histograms/) for more information. + + Use one of the following strategies to update dashboards. + + - (Recommended) Add new dashboards with the new exponential histogram queries. This solution requires looking at different dashboards for data before and after the migration, until data before the migration is removed due to passing its retention time. You can publish the new dashboard when sufficient time has passed to serve users with the new data. + - Add a dashboard variable to your dashboard to enable switching between explicit bucket histograms and exponential histograms. There isn't support for selectively enabling and disabling queries in Grafana ([issue 79848](https://github.com/grafana/grafana/issues/79848)). As a workaround, add the dashboard variable `latency_metrics`, for example, and assign it a value of either `-1` or `1`. Then, add the following two queries to the panel: + + ``` + < ($latency_metrics * +Inf) + ``` + + ``` + < ($latency_metrics * -Inf) + ``` + + Where `classic_query` is the original query and `native_query` is the same query using exponential histogram query syntax. This technique is employed in Mimir's dashboards. For an example, refer to the [Overview dashboard](https://github.com/grafana/mimir/blob/main/operations/mimir-mixin-compiled/dashboards/mimir-overview.json) in the Mimir repository. + + This solution allows users to switch between the explicit bucket histogram and the exponential histogram without going to a different dashboard. + + - Replace the existing explicit bucket queries with modified queries. For example, replace: + + ``` + + ``` + + with + + ``` + or + ``` + + Where `explicit_bucket_query` is the original query and `exponential_query` is the same query using exponential histogram query syntax. + + {{< admonition type="warning" >}} + Using the PromQL operator `or` can lead to unexpected results. For example, if a query uses a range of seven days, such as `sum(rate(http_request_duration_seconds[7d]))`, then this query returns a value as soon as there are two exponential histograms samples present before the end time specified in the query. In this case, the seven day rate is calculated from a couple of minutes, rather than seven days, worth of data. This results in an inaccuracy in the graph around the time you started scraping exponential histograms. + {{< /admonition >}} + +1. Start adding new recording rules and alerts to use exponential histograms. Don't remove the old recording rules and alerts at this time. +1. It's important to keep scraping both explicit bucket and exponential histograms for at least the period of the longest range in your recording rules and alerts, plus one day. This is the minimum amount of time, but it's recommended to keep scraping both data types until you can verify the new rules and alerts. + + For example, if you have an alert that calculates the rate of requests, such as `sum(rate(http_request_duration_seconds[7d]))`, this query looks at the data from the last seven days plus the Prometheus [lookback period](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness). When you start sending exponential histograms, the data isn't there for the entire seven days, and therefore, the results might be unreliable for alerting. + +1. After configuring exponential histogram collection, choose one of the following ways to stop collecting explicit bucket histograms. + + - Remove the custom bucket definition, `Buckets`/`classicUpperBounds`, from the instrumentation. In Java, also use the `nativeOnly()` option. Refer to the examples in [Instrument application with Prometheus client libraries](#instrument-application-with-prometheus-client-libraries). + - Drop the explicit bucket histogram series with [Prometheus relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) or [Grafana Alloy prometheus.relabel](https://grafana.com/docs/alloy//reference/components/prometheus/prometheus.relabel) at the time of scraping. + - Stop scraping the explicit bucket histogram version of metrics. This option applies to all metrics of a scrape target. + +1. Clean up recording rules and alerts by deleting the explicit bucket histogram version of the rule or alert. ## Bucket boundary calculation \ No newline at end of file From e52640d8853002f95d574b18ce7d3854fbc44711 Mon Sep 17 00:00:00 2001 From: Taylor Cole Date: Mon, 23 Sep 2024 17:30:25 -0700 Subject: [PATCH 10/16] add bucket boundary calculation info for exp histograms --- .../mimir/send/otel-exponential-histograms/_index.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/sources/mimir/send/otel-exponential-histograms/_index.md b/docs/sources/mimir/send/otel-exponential-histograms/_index.md index 702221e5427..10f8ad68e28 100644 --- a/docs/sources/mimir/send/otel-exponential-histograms/_index.md +++ b/docs/sources/mimir/send/otel-exponential-histograms/_index.md @@ -111,4 +111,12 @@ To ease the migration process, you can keep the custom bucket definition of an e 1. Clean up recording rules and alerts by deleting the explicit bucket histogram version of the rule or alert. -## Bucket boundary calculation \ No newline at end of file +## Bucket boundary calculation + +Bucket boundaries for exponential histograms are calculated similarly to those for native histograms. The only difference is that for exponential histograms, bucket offsets are shifted by one. + +For example: + + + +For more information, refer to [bucket boundary calculation](https://grafana.com/docs/mimir/next/send/native-histograms/#bucket-boundary-calculation) in the documentation for native histograms. From 040368d4e8a8907f2efed0762d64feaba5033286 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gy=C3=B6rgy=20Krajcsovits?= Date: Tue, 24 Sep 2024 10:08:33 +0200 Subject: [PATCH 11/16] Add otel bucket definition and remove stray line MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: György Krajcsovits --- .../sources/mimir/send/otel-exponential-histograms/_index.md | 5 +++-- .../send/otel-exponential-histograms/otel-pos-bucket-def.svg | 2 ++ 2 files changed, 5 insertions(+), 2 deletions(-) create mode 100644 docs/sources/mimir/send/otel-exponential-histograms/otel-pos-bucket-def.svg diff --git a/docs/sources/mimir/send/otel-exponential-histograms/_index.md b/docs/sources/mimir/send/otel-exponential-histograms/_index.md index 10f8ad68e28..8e44a39cfd6 100644 --- a/docs/sources/mimir/send/otel-exponential-histograms/_index.md +++ b/docs/sources/mimir/send/otel-exponential-histograms/_index.md @@ -56,7 +56,6 @@ To ease the migration process, you can keep the custom bucket definition of an e }, sdkmetric.Stream{ Name: "request_latency_exp", Aggregation: sdkmetric.AggregationBase2ExponentialHistogram{MaxSize: 160, NoMinMax: true, MaxScale: 20}, - //Aggregation: sdkmetric.DefaultAggregationSelector(sdkmetric.InstrumentKindHistogram), }) ``` @@ -115,8 +114,10 @@ To ease the migration process, you can keep the custom bucket definition of an e Bucket boundaries for exponential histograms are calculated similarly to those for native histograms. The only difference is that for exponential histograms, bucket offsets are shifted by one. -For example: +Thus: + +![Positive bucket definition](otel-pos-bucket-def.svg) For more information, refer to [bucket boundary calculation](https://grafana.com/docs/mimir/next/send/native-histograms/#bucket-boundary-calculation) in the documentation for native histograms. diff --git a/docs/sources/mimir/send/otel-exponential-histograms/otel-pos-bucket-def.svg b/docs/sources/mimir/send/otel-exponential-histograms/otel-pos-bucket-def.svg new file mode 100644 index 00000000000..0506bb3306d --- /dev/null +++ b/docs/sources/mimir/send/otel-exponential-histograms/otel-pos-bucket-def.svg @@ -0,0 +1,2 @@ + + \ No newline at end of file From 59c3f35326be4ee650c79fefba1d7c239e2ad73c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gy=C3=B6rgy=20Krajcsovits?= Date: Tue, 24 Sep 2024 10:13:03 +0200 Subject: [PATCH 12/16] ran make docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: György Krajcsovits --- .../otel-exponential-histograms/_index.md | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/sources/mimir/send/otel-exponential-histograms/_index.md b/docs/sources/mimir/send/otel-exponential-histograms/_index.md index 8e44a39cfd6..a37f3a1beae 100644 --- a/docs/sources/mimir/send/otel-exponential-histograms/_index.md +++ b/docs/sources/mimir/send/otel-exponential-histograms/_index.md @@ -12,7 +12,7 @@ weight: 200 # Send OpenTelemetry exponential histograms to Mimir -You can collect and send exponential histograms to Mimir with the OpenTelemetry Collector. OpenTelemetry [exponential histograms](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponentialhistogram) are compatible with Prometheus native histograms. The key difference is that exponential histograms store the `min` and `max` observation values explicitly, whereas native histograms don't. This means that for exponential histograms, you don't need to estimate these values using the 0.0 and 1.0 quantiles. +You can collect and send exponential histograms to Mimir with the OpenTelemetry Collector. OpenTelemetry [exponential histograms](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponentialhistogram) are compatible with Prometheus native histograms. The key difference is that exponential histograms store the `min` and `max` observation values explicitly, whereas native histograms don't. This means that for exponential histograms, you don't need to estimate these values using the 0.0 and 1.0 quantiles. The OpenTelemetry Collector supports collecting exponential histograms and other compatible data formats, such as native histograms and DataDog sketches, through its receivers and sending them through its exporters. @@ -22,7 +22,7 @@ The availability of different receivers and exporters depends on your OpenTeleme You can use the OpenTelemetry (OTLP) protocol to send exponential histograms to Grafana Mimir in their existing format, or you can use the Prometheus remote write protocol to send them as Prometheus native histograms. Refer to [Send native histograms to Mimir](https://grafana.com/docs/mimir//send/native-histograms/). -The OpenTelemetry SDK supports instrumenting applications in multiple languages. Refer to [Language APIs & SDKs](https://opentelemetry.io/docs/languages/). +The OpenTelemetry SDK supports instrumenting applications in multiple languages. Refer to [Language APIs & SDKs](https://opentelemetry.io/docs/languages/). ## Instrument an application with the OpenTelemetry SDK using Go @@ -33,9 +33,9 @@ Use the OpenTelemetry SDK version 1.17.0 or later. ``` Aggregation: metric.AggregationBase2ExponentialHistogram{ - MaxSize: 160, - MaxScale: 20, - } + MaxSize: 160, + MaxScale: 20, + } ``` For more information about views, refer to [Registering Views](https://opentelemetry.io/docs/languages/go/instrumentation/#registering-views) in the OpenTelemetry SDK documentation for Go. For information about view configuration parameters, refer to [Base2 Exponential Bucket Histogram Aggregation](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk.md#base2-exponential-bucket-histogram-aggregation) in the OpenTelemetry Metrics SDK on GitHub. @@ -47,19 +47,20 @@ To ease the migration process, you can keep the custom bucket definition of an e 1. Start with an existing histogram that uses explicit buckets. 1. Create a view for the exponential histogram. Assign this view a unique name and include the exponential aggregation. This creates a metric with the assigned name and exponential buckets. - The following example shows how to create a metric called `request_latency_exp` that uses exponential buckets. +The following example shows how to create a metric called `request_latency_exp` that uses exponential buckets. - ``` - v := sdkmetric.NewView(sdkmetric.Instrument{ +``` +v := sdkmetric.NewView(sdkmetric.Instrument{ Name: "request_latency", Kind: sdkmetric.InstrumentKindHistogram, }, sdkmetric.Stream{ Name: "request_latency_exp", Aggregation: sdkmetric.AggregationBase2ExponentialHistogram{MaxSize: 160, NoMinMax: true, MaxScale: 20}, }) - ``` +``` + +For more information about creating a view, refer to [Views](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#view) in the OpenTelemetry Metrics SDK. - For more information about creating a view, refer to [Views](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#view) in the OpenTelemetry Metrics SDK. 1. Modify dashboards to use the exponential histogram metrics. Refer to [Visualize native histograms](https://grafana.com/docs/mimir//visualize/native-histograms/) for more information. Use one of the following strategies to update dashboards. From 887f28fd0579c365f08f8660ffe9d4e9a1b4eb7c Mon Sep 17 00:00:00 2001 From: Taylor Cole Date: Tue, 24 Sep 2024 15:05:49 -0700 Subject: [PATCH 13/16] docs: add feedback for exponential histograms --- .../mimir/send/otel-exponential-histograms/_index.md | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/docs/sources/mimir/send/otel-exponential-histograms/_index.md b/docs/sources/mimir/send/otel-exponential-histograms/_index.md index a37f3a1beae..fc0296ad207 100644 --- a/docs/sources/mimir/send/otel-exponential-histograms/_index.md +++ b/docs/sources/mimir/send/otel-exponential-histograms/_index.md @@ -103,12 +103,7 @@ For more information about creating a view, refer to [Views](https://opentelemet For example, if you have an alert that calculates the rate of requests, such as `sum(rate(http_request_duration_seconds[7d]))`, this query looks at the data from the last seven days plus the Prometheus [lookback period](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness). When you start sending exponential histograms, the data isn't there for the entire seven days, and therefore, the results might be unreliable for alerting. -1. After configuring exponential histogram collection, choose one of the following ways to stop collecting explicit bucket histograms. - - - Remove the custom bucket definition, `Buckets`/`classicUpperBounds`, from the instrumentation. In Java, also use the `nativeOnly()` option. Refer to the examples in [Instrument application with Prometheus client libraries](#instrument-application-with-prometheus-client-libraries). - - Drop the explicit bucket histogram series with [Prometheus relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) or [Grafana Alloy prometheus.relabel](https://grafana.com/docs/alloy//reference/components/prometheus/prometheus.relabel) at the time of scraping. - - Stop scraping the explicit bucket histogram version of metrics. This option applies to all metrics of a scrape target. - +1. After configuring exponential histogram collection, remove the explicit bucket histogram definition, as well as any views that expose explicit buckets. 1. Clean up recording rules and alerts by deleting the explicit bucket histogram version of the rule or alert. ## Bucket boundary calculation From 2f6fcabddce90ee01abd33a9b5b3dd11be315e68 Mon Sep 17 00:00:00 2001 From: Taylor Cole Date: Tue, 24 Sep 2024 15:45:56 -0700 Subject: [PATCH 14/16] docs: add feedback for exponential histograms --- .../otel-exponential-histograms/_index.md | 44 +++++++++---------- 1 file changed, 21 insertions(+), 23 deletions(-) diff --git a/docs/sources/mimir/send/otel-exponential-histograms/_index.md b/docs/sources/mimir/send/otel-exponential-histograms/_index.md index fc0296ad207..2e1f7e102b8 100644 --- a/docs/sources/mimir/send/otel-exponential-histograms/_index.md +++ b/docs/sources/mimir/send/otel-exponential-histograms/_index.md @@ -14,15 +14,15 @@ weight: 200 You can collect and send exponential histograms to Mimir with the OpenTelemetry Collector. OpenTelemetry [exponential histograms](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponentialhistogram) are compatible with Prometheus native histograms. The key difference is that exponential histograms store the `min` and `max` observation values explicitly, whereas native histograms don't. This means that for exponential histograms, you don't need to estimate these values using the 0.0 and 1.0 quantiles. -The OpenTelemetry Collector supports collecting exponential histograms and other compatible data formats, such as native histograms and DataDog sketches, through its receivers and sending them through its exporters. +The OpenTelemetry Collector supports collecting exponential histograms and other compatible data formats, including native histograms and Datadog sketches, through its receivers and sending them through its exporters. {{< admonition type="note" >}} The availability of different receivers and exporters depends on your OpenTelemetry Collector [distribution](https://opentelemetry.io/docs/concepts/distributions/). {{< /admonition >}} -You can use the OpenTelemetry (OTLP) protocol to send exponential histograms to Grafana Mimir in their existing format, or you can use the Prometheus remote write protocol to send them as Prometheus native histograms. Refer to [Send native histograms to Mimir](https://grafana.com/docs/mimir//send/native-histograms/). +You can use the OpenTelemetry protocol (OTLP) to send exponential histograms to Grafana Mimir in their existing format, or you can use the Prometheus remote write protocol to send them as Prometheus native histograms. Refer to [Send native histograms to Mimir](https://grafana.com/docs/mimir//send/native-histograms/). -The OpenTelemetry SDK supports instrumenting applications in multiple languages. Refer to [Language APIs & SDKs](https://opentelemetry.io/docs/languages/). +The OpenTelemetry SDK supports instrumenting applications in multiple languages. Refer to [Language APIs & SDKs](https://opentelemetry.io/docs/languages/) for a complete list. ## Instrument an application with the OpenTelemetry SDK using Go @@ -47,40 +47,40 @@ To ease the migration process, you can keep the custom bucket definition of an e 1. Start with an existing histogram that uses explicit buckets. 1. Create a view for the exponential histogram. Assign this view a unique name and include the exponential aggregation. This creates a metric with the assigned name and exponential buckets. -The following example shows how to create a metric called `request_latency_exp` that uses exponential buckets. + The following example shows how to create a metric called `request_latency_exp` that uses exponential buckets. -``` -v := sdkmetric.NewView(sdkmetric.Instrument{ - Name: "request_latency", - Kind: sdkmetric.InstrumentKindHistogram, - }, sdkmetric.Stream{ - Name: "request_latency_exp", - Aggregation: sdkmetric.AggregationBase2ExponentialHistogram{MaxSize: 160, NoMinMax: true, MaxScale: 20}, - }) -``` + ``` + v := sdkmetric.NewView(sdkmetric.Instrument{ + Name: "request_latency", + Kind: sdkmetric.InstrumentKindHistogram, + }, sdkmetric.Stream{ + Name: "request_latency_exp", + Aggregation: sdkmetric.AggregationBase2ExponentialHistogram{MaxSize: 160, NoMinMax: true, MaxScale: 20}, + }) + ``` -For more information about creating a view, refer to [Views](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#view) in the OpenTelemetry Metrics SDK. + For more information about creating a view, refer to [View](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#view) in the OpenTelemetry Metrics SDK. 1. Modify dashboards to use the exponential histogram metrics. Refer to [Visualize native histograms](https://grafana.com/docs/mimir//visualize/native-histograms/) for more information. Use one of the following strategies to update dashboards. - - (Recommended) Add new dashboards with the new exponential histogram queries. This solution requires looking at different dashboards for data before and after the migration, until data before the migration is removed due to passing its retention time. You can publish the new dashboard when sufficient time has passed to serve users with the new data. + - (Recommended) Create dashboards with the exponential histogram queries. This solution requires looking at different dashboards for data before and after the migration, until data before the migration is removed due to passing its retention time. You can publish the dashboard when sufficient time has passed to serve users with the new data. - Add a dashboard variable to your dashboard to enable switching between explicit bucket histograms and exponential histograms. There isn't support for selectively enabling and disabling queries in Grafana ([issue 79848](https://github.com/grafana/grafana/issues/79848)). As a workaround, add the dashboard variable `latency_metrics`, for example, and assign it a value of either `-1` or `1`. Then, add the following two queries to the panel: ``` - < ($latency_metrics * +Inf) + < ($latency_metrics * +Inf) ``` ``` - < ($latency_metrics * -Inf) + < ($latency_metrics * -Inf) ``` - Where `classic_query` is the original query and `native_query` is the same query using exponential histogram query syntax. This technique is employed in Mimir's dashboards. For an example, refer to the [Overview dashboard](https://github.com/grafana/mimir/blob/main/operations/mimir-mixin-compiled/dashboards/mimir-overview.json) in the Mimir repository. + Where `explicit_bucket_query` is the original query and `exponential_query` is the same query using exponential histogram query syntax. This technique is employed in Mimir's dashboards. For an example, refer to the [Overview dashboard](https://github.com/grafana/mimir/blob/main/operations/mimir-mixin-compiled/dashboards/mimir-overview.json) in the Mimir repository. This solution allows users to switch between the explicit bucket histogram and the exponential histogram without going to a different dashboard. - - Replace the existing explicit bucket queries with modified queries. For example, replace: + - Replace the explicit bucket queries with modified queries. For example, replace: ``` @@ -98,7 +98,7 @@ For more information about creating a view, refer to [Views](https://opentelemet Using the PromQL operator `or` can lead to unexpected results. For example, if a query uses a range of seven days, such as `sum(rate(http_request_duration_seconds[7d]))`, then this query returns a value as soon as there are two exponential histograms samples present before the end time specified in the query. In this case, the seven day rate is calculated from a couple of minutes, rather than seven days, worth of data. This results in an inaccuracy in the graph around the time you started scraping exponential histograms. {{< /admonition >}} -1. Start adding new recording rules and alerts to use exponential histograms. Don't remove the old recording rules and alerts at this time. +1. Begin adding recording rules and alerts to use exponential histograms. Don't remove any existing recording rules and alerts at this time. 1. It's important to keep scraping both explicit bucket and exponential histograms for at least the period of the longest range in your recording rules and alerts, plus one day. This is the minimum amount of time, but it's recommended to keep scraping both data types until you can verify the new rules and alerts. For example, if you have an alert that calculates the rate of requests, such as `sum(rate(http_request_duration_seconds[7d]))`, this query looks at the data from the last seven days plus the Prometheus [lookback period](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness). When you start sending exponential histograms, the data isn't there for the entire seven days, and therefore, the results might be unreliable for alerting. @@ -108,9 +108,7 @@ For more information about creating a view, refer to [Views](https://opentelemet ## Bucket boundary calculation -Bucket boundaries for exponential histograms are calculated similarly to those for native histograms. The only difference is that for exponential histograms, bucket offsets are shifted by one. - -Thus: +Bucket boundaries for exponential histograms are calculated similarly to those for native histograms. The only difference is that for exponential histograms, bucket offsets are shifted by one, as shown in the following equation. From 254f484fc0701de03fab260bdb45ecff42cb8101 Mon Sep 17 00:00:00 2001 From: Taylor C <41653732+tacole02@users.noreply.github.com> Date: Wed, 25 Sep 2024 15:44:06 -0700 Subject: [PATCH 15/16] Update docs/sources/mimir/send/otel-exponential-histograms/_index.md Co-authored-by: George Krajcsovits --- docs/sources/mimir/send/otel-exponential-histograms/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/mimir/send/otel-exponential-histograms/_index.md b/docs/sources/mimir/send/otel-exponential-histograms/_index.md index 2e1f7e102b8..3c1d37661ac 100644 --- a/docs/sources/mimir/send/otel-exponential-histograms/_index.md +++ b/docs/sources/mimir/send/otel-exponential-histograms/_index.md @@ -20,7 +20,7 @@ The OpenTelemetry Collector supports collecting exponential histograms and other The availability of different receivers and exporters depends on your OpenTelemetry Collector [distribution](https://opentelemetry.io/docs/concepts/distributions/). {{< /admonition >}} -You can use the OpenTelemetry protocol (OTLP) to send exponential histograms to Grafana Mimir in their existing format, or you can use the Prometheus remote write protocol to send them as Prometheus native histograms. Refer to [Send native histograms to Mimir](https://grafana.com/docs/mimir//send/native-histograms/). +You can use the OpenTelemetry protocol (OTLP) over HTTP to send exponential histograms to Grafana Mimir in their existing format, or you can use the Prometheus remote write protocol to send them as Prometheus native histograms. The OpenTelemetry SDK supports instrumenting applications in multiple languages. Refer to [Language APIs & SDKs](https://opentelemetry.io/docs/languages/) for a complete list. From 94384dbe33f9fd68039195b22ede025c8c7c9d80 Mon Sep 17 00:00:00 2001 From: Taylor Cole Date: Thu, 26 Sep 2024 12:54:52 -0700 Subject: [PATCH 16/16] docs: make doc output --- .../mimir/send/otel-exponential-histograms/_index.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/sources/mimir/send/otel-exponential-histograms/_index.md b/docs/sources/mimir/send/otel-exponential-histograms/_index.md index 3c1d37661ac..22382fafb28 100644 --- a/docs/sources/mimir/send/otel-exponential-histograms/_index.md +++ b/docs/sources/mimir/send/otel-exponential-histograms/_index.md @@ -51,12 +51,12 @@ To ease the migration process, you can keep the custom bucket definition of an e ``` v := sdkmetric.NewView(sdkmetric.Instrument{ - Name: "request_latency", - Kind: sdkmetric.InstrumentKindHistogram, - }, sdkmetric.Stream{ - Name: "request_latency_exp", - Aggregation: sdkmetric.AggregationBase2ExponentialHistogram{MaxSize: 160, NoMinMax: true, MaxScale: 20}, - }) + Name: "request_latency", + Kind: sdkmetric.InstrumentKindHistogram, + }, sdkmetric.Stream{ + Name: "request_latency_exp", + Aggregation: sdkmetric.AggregationBase2ExponentialHistogram{MaxSize: 160, NoMinMax: true, MaxScale: 20}, + }) ``` For more information about creating a view, refer to [View](https://opentelemetry.io/docs/specs/otel/metrics/sdk/#view) in the OpenTelemetry Metrics SDK.