From bfd33d9af5b9060b96151dc4faaa2122c8e17c2d Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Wed, 28 Feb 2024 18:11:18 -0800 Subject: [PATCH 01/30] Creating "Composite Samplers" OTEP. --- text/0000-Composite_Samplers.md | 118 ++++++++++++++++++++++++++++++++ 1 file changed, 118 insertions(+) create mode 100644 text/0000-Composite_Samplers.md diff --git a/text/0000-Composite_Samplers.md b/text/0000-Composite_Samplers.md new file mode 100644 index 000000000..2de373aa7 --- /dev/null +++ b/text/0000-Composite_Samplers.md @@ -0,0 +1,118 @@ +# Composite Samplers + +This proposal addresses head-based sampling as described by the [Open Telemetry SDK](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#sampling). +It introduces additional composite samplers. Composite samplers use other samplers (delegates) to make sampling decisions. The composite samplers invoke the delegate samplers, but eventually make the final call. + +## Motivation + +The need for configuring head sampling has been explicitly or implicitly indicated in several discussions, both within the [Samplig SIG](https://docs.google.com/document/d/1gASMhmxNt9qCa8czEMheGlUW2xpORiYoD7dBD7aNtbQ) and in the wider community. Some of the discussions are going back a number of years, see for example + +- issue [173](https://github.com/open-telemetry/opentelemetry-specification/issues/173): Way to ignore healthcheck traces when using automatic tracer across all languages? +- issue [1060](https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/1060): Exclude URLs from Tracing +- issue [1844](https://github.com/open-telemetry/opentelemetry-specification/issues/1844): Composite Sampler + +## The Goal + +The goal of this proposal is to extend the set of standard SDK samplers by _Composite Samplers_ which make building advanced sampling configuration easier. + +### Example + +Let's assume that a user wants to configure head sampling as follows: + +- for root spans: + - drop all `/healthcheck` requests + - capture all `/checkout` requests + - capture 25% of all other requests +- for non-root spans + - follow the parent sampling decision + - however, capture all calls to service `/foo` (even if the trace will be incomplete) +- in any case, do not exceed 1000 spans/minute + +It is hoped that the Composite Samplers presented here will be useful in constructing a sampling configuration satisfying the above case. + +## AnyOf + +`AnyOf` is a composite sampler which takes a non-empty list of Samplers (delegates) as the argument. + +Upon invocation of its `shouldSample` method, it MUST go through the whole list and invoke `shouldSample` method on each delegate sampler, passing the same arguments as received. + +`AnyOf` sampler MUST return a `SamplingResult` which is constructed as follows: + +- The sampling Decision is based on the delegate sampling Decisions. If all of the delegate Decisions are `DROP`, the composite sampler MUST return `DROP` Decision as well. +If any of the delegate Decisions is `RECORD_AND_SAMPLE`, the composite sampler MUST return `RECORD_AND_SAMPLE` Decision. +Otherwise, if any of the delegate Decisions is `RECORD_ONLY`, the composite sampler MUST return `RECORD_ONLY` Decision. +- The set of span Attributes to be added to the `Span` is the sum of the sets of Attributes as provided by the delegate samplers within their `SamplingResults`s. +- The `TraceState` to be used with the new `Span` is obtained by cumulatively applying all the potential modfications of the parent `TraceState` by the delegate samplers. + +Each delegate sampler MUST be given a chance to participate in the sampling decision as described above and MUST see the same _parent_ state. The order of the delegate samplers does not matter, as long as there's no overlap in the Attribute Keys or the trace state keys they use. + +## Conjunction + +`Conjunction` is a composite sampler which takes two Samplers (delegates) as the arguments. These delegate samplers will be hereby referenced as First and Second. + +Upon invocation of its `shouldSample` method, the Conjunction sampler MUST invoke `shouldSample` method on the First sampler, passing the same arguments as received, and examine the received sampling Decision. Upon receiving `DROP` or `RECORD_ONLY` decision it MUST return the SamplingResult from the First sampler, and it MUST NOT proceed with querying the Second sampler. If the sampling decision from the First sampler is `RECORD_AND_SAMPLE`, the Conjunction sampler MUST invoke `shouldSample` method on the Second sampler. +If the sampling Decision from the Second sampler is `RECORD_AND_SAMPLE`, the Conjunction sampler MUST return a `SamplingResult` which is constructed as follows: + +- The sampling Decision is `RECORD_AND_SAMPLE`. +- The set of span Attributes to be added to the `Span` is the sum of the sets of Attributes as provided by the First and the Second sampler. +- The `TraceState` to be used with the new `Span` is obtained by cumulatively applying the potential modfications from the First and Second sampler. + +If the sampling Decision from the Second sampler is `RECORD_ONLY` or `DROP`, the Conjunction sampler MUST retrun a `SamplingResult` which is constructed as follows: + +- The sampling Decision is `DROP`. +- The set of span Attributes to be added to the `Span` is empty. +- The `TraceState` to be used with the new `Span` is the passed-in `TraceState`. + + +## Rule based sampling + +For rule-based sampling (e.g. when sampling decision should depend on Span attributes), the Spans can be grouped into separate categories, and each category can use a different Sampler. Categorization of Spans is aided by `Predicates`. +The Predicates represent logical expressions which can access Span Attributes (or anything else available when the sampling decision is to be made), and perform tests on the accessible values. +For example, one can test if the target URL for a SERVER span matches a given pattern. + +### Predicate + +`Predicate` interface allows users to create custom categories based on information that is availabe at the time of making the sampling decision. + +#### SpanMatches + +This is a function/method for `Predicate`, which returns `true` if a given `Span` matches, i.e. belongs to the category described by the Predicate. + +#### Required Arguments + +The arguments represent the values that are made availabe for `ShouldSample`. + +- `Context` with parent Span. +- `TraceId` of the Span to be created. +- Name of the Span to be created. +- Initial set of Attributes of the Span to be created. +- Collection of links that will be associated with the Span to be created. + +## RuleBased + +`RuleBased` is a composite sampler which performs `Span` categorization and sampling. +It takes the following arguments: +- `SpanKind` +- list of `Predicate`s +- list of `Sampler`s + +The `RuleBased` sampler SHOULD NOT accept lists of different length, i.e. it SHOULD report an error. Implementations MAY allow for replacing both lists with a list of pairs (`Predicate`, `Sampler`), if this is supported by the platform. + +For making the sampling decision, if the `Span` kind matches the specified kind, the sampler goes through the lists in the declared order. If a Predicate matches the `Span` in question, the corresponding `Sampler` will be called to make the final sampling decision. If the `SpanKind` does not match, or none of the Predicates evaluates to `true`, the final decision is `DROP`. + +## Limitations of composite samplers + +Not all samplers can participate as components of composite samplers without undesired or unexpected effects. Some samplers require that they _see_ each `Span` being created, even if the span is going to be dropped. Some samplers update the trace state or maintain internal state, and for their correct behavior it it is assumed that their sampling decisions will be honored by the tracer at the face value in all cases. A good example for this are rate limiting samplers which have to keep track of the rate of created spans and/or the rate of positive sampling decisions. + +Placing such samplers as the first or second argument for the `Conjunction` sampler can sometimes preserve their correct behavior. + +A special attention is required for [consistent probability](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#consistent-probability-sampler) (CP) samplers. The sampling probability they record in trace-state is later used to calculate [_adjusted count_](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#adjusted-count), which, in turn, is used to calculate span-based metrics. Mixing CP samplers with other types of samplers in most cases will lead to incorrect adjusted counts. The family of CP samplers has its own [composition rules](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#composition-rules), which correctly handle composing multiple CP samplers. + +## Prior art + +A number of composite samplers are already available as independent contributions +([RuleBasedRoutingSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/RuleBasedRoutingSampler.java), +[LinksBasedSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/LinksBasedSampler.java)). +Also, historically, some Span categorization was introduced by [JaegerRemoteSampler](https://www.jaegertracing.io/docs/1.54/sampling/#remote-sampling). + +This proposal aims at generalizing these ideas, and at providing a bit more formal specification for the behavior of the composite samplers. From 611a1401b72ae6e222c1bbe4e03bee0b2110e22d Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Wed, 28 Feb 2024 18:22:39 -0800 Subject: [PATCH 02/30] File rename --- text/{0000-Composite_Samplers.md => 0250-Composite_Samplers.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename text/{0000-Composite_Samplers.md => 0250-Composite_Samplers.md} (100%) diff --git a/text/0000-Composite_Samplers.md b/text/0250-Composite_Samplers.md similarity index 100% rename from text/0000-Composite_Samplers.md rename to text/0250-Composite_Samplers.md From 772ef286e2d7b56ce83087c02c8332a9b13d7827 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Wed, 28 Feb 2024 18:11:18 -0800 Subject: [PATCH 03/30] Creating "Composite Samplers" OTEP. --- text/0000-Composite_Samplers.md | 118 ++++++++++++++++++++++++++++++++ 1 file changed, 118 insertions(+) create mode 100644 text/0000-Composite_Samplers.md diff --git a/text/0000-Composite_Samplers.md b/text/0000-Composite_Samplers.md new file mode 100644 index 000000000..2de373aa7 --- /dev/null +++ b/text/0000-Composite_Samplers.md @@ -0,0 +1,118 @@ +# Composite Samplers + +This proposal addresses head-based sampling as described by the [Open Telemetry SDK](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#sampling). +It introduces additional composite samplers. Composite samplers use other samplers (delegates) to make sampling decisions. The composite samplers invoke the delegate samplers, but eventually make the final call. + +## Motivation + +The need for configuring head sampling has been explicitly or implicitly indicated in several discussions, both within the [Samplig SIG](https://docs.google.com/document/d/1gASMhmxNt9qCa8czEMheGlUW2xpORiYoD7dBD7aNtbQ) and in the wider community. Some of the discussions are going back a number of years, see for example + +- issue [173](https://github.com/open-telemetry/opentelemetry-specification/issues/173): Way to ignore healthcheck traces when using automatic tracer across all languages? +- issue [1060](https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/1060): Exclude URLs from Tracing +- issue [1844](https://github.com/open-telemetry/opentelemetry-specification/issues/1844): Composite Sampler + +## The Goal + +The goal of this proposal is to extend the set of standard SDK samplers by _Composite Samplers_ which make building advanced sampling configuration easier. + +### Example + +Let's assume that a user wants to configure head sampling as follows: + +- for root spans: + - drop all `/healthcheck` requests + - capture all `/checkout` requests + - capture 25% of all other requests +- for non-root spans + - follow the parent sampling decision + - however, capture all calls to service `/foo` (even if the trace will be incomplete) +- in any case, do not exceed 1000 spans/minute + +It is hoped that the Composite Samplers presented here will be useful in constructing a sampling configuration satisfying the above case. + +## AnyOf + +`AnyOf` is a composite sampler which takes a non-empty list of Samplers (delegates) as the argument. + +Upon invocation of its `shouldSample` method, it MUST go through the whole list and invoke `shouldSample` method on each delegate sampler, passing the same arguments as received. + +`AnyOf` sampler MUST return a `SamplingResult` which is constructed as follows: + +- The sampling Decision is based on the delegate sampling Decisions. If all of the delegate Decisions are `DROP`, the composite sampler MUST return `DROP` Decision as well. +If any of the delegate Decisions is `RECORD_AND_SAMPLE`, the composite sampler MUST return `RECORD_AND_SAMPLE` Decision. +Otherwise, if any of the delegate Decisions is `RECORD_ONLY`, the composite sampler MUST return `RECORD_ONLY` Decision. +- The set of span Attributes to be added to the `Span` is the sum of the sets of Attributes as provided by the delegate samplers within their `SamplingResults`s. +- The `TraceState` to be used with the new `Span` is obtained by cumulatively applying all the potential modfications of the parent `TraceState` by the delegate samplers. + +Each delegate sampler MUST be given a chance to participate in the sampling decision as described above and MUST see the same _parent_ state. The order of the delegate samplers does not matter, as long as there's no overlap in the Attribute Keys or the trace state keys they use. + +## Conjunction + +`Conjunction` is a composite sampler which takes two Samplers (delegates) as the arguments. These delegate samplers will be hereby referenced as First and Second. + +Upon invocation of its `shouldSample` method, the Conjunction sampler MUST invoke `shouldSample` method on the First sampler, passing the same arguments as received, and examine the received sampling Decision. Upon receiving `DROP` or `RECORD_ONLY` decision it MUST return the SamplingResult from the First sampler, and it MUST NOT proceed with querying the Second sampler. If the sampling decision from the First sampler is `RECORD_AND_SAMPLE`, the Conjunction sampler MUST invoke `shouldSample` method on the Second sampler. +If the sampling Decision from the Second sampler is `RECORD_AND_SAMPLE`, the Conjunction sampler MUST return a `SamplingResult` which is constructed as follows: + +- The sampling Decision is `RECORD_AND_SAMPLE`. +- The set of span Attributes to be added to the `Span` is the sum of the sets of Attributes as provided by the First and the Second sampler. +- The `TraceState` to be used with the new `Span` is obtained by cumulatively applying the potential modfications from the First and Second sampler. + +If the sampling Decision from the Second sampler is `RECORD_ONLY` or `DROP`, the Conjunction sampler MUST retrun a `SamplingResult` which is constructed as follows: + +- The sampling Decision is `DROP`. +- The set of span Attributes to be added to the `Span` is empty. +- The `TraceState` to be used with the new `Span` is the passed-in `TraceState`. + + +## Rule based sampling + +For rule-based sampling (e.g. when sampling decision should depend on Span attributes), the Spans can be grouped into separate categories, and each category can use a different Sampler. Categorization of Spans is aided by `Predicates`. +The Predicates represent logical expressions which can access Span Attributes (or anything else available when the sampling decision is to be made), and perform tests on the accessible values. +For example, one can test if the target URL for a SERVER span matches a given pattern. + +### Predicate + +`Predicate` interface allows users to create custom categories based on information that is availabe at the time of making the sampling decision. + +#### SpanMatches + +This is a function/method for `Predicate`, which returns `true` if a given `Span` matches, i.e. belongs to the category described by the Predicate. + +#### Required Arguments + +The arguments represent the values that are made availabe for `ShouldSample`. + +- `Context` with parent Span. +- `TraceId` of the Span to be created. +- Name of the Span to be created. +- Initial set of Attributes of the Span to be created. +- Collection of links that will be associated with the Span to be created. + +## RuleBased + +`RuleBased` is a composite sampler which performs `Span` categorization and sampling. +It takes the following arguments: +- `SpanKind` +- list of `Predicate`s +- list of `Sampler`s + +The `RuleBased` sampler SHOULD NOT accept lists of different length, i.e. it SHOULD report an error. Implementations MAY allow for replacing both lists with a list of pairs (`Predicate`, `Sampler`), if this is supported by the platform. + +For making the sampling decision, if the `Span` kind matches the specified kind, the sampler goes through the lists in the declared order. If a Predicate matches the `Span` in question, the corresponding `Sampler` will be called to make the final sampling decision. If the `SpanKind` does not match, or none of the Predicates evaluates to `true`, the final decision is `DROP`. + +## Limitations of composite samplers + +Not all samplers can participate as components of composite samplers without undesired or unexpected effects. Some samplers require that they _see_ each `Span` being created, even if the span is going to be dropped. Some samplers update the trace state or maintain internal state, and for their correct behavior it it is assumed that their sampling decisions will be honored by the tracer at the face value in all cases. A good example for this are rate limiting samplers which have to keep track of the rate of created spans and/or the rate of positive sampling decisions. + +Placing such samplers as the first or second argument for the `Conjunction` sampler can sometimes preserve their correct behavior. + +A special attention is required for [consistent probability](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#consistent-probability-sampler) (CP) samplers. The sampling probability they record in trace-state is later used to calculate [_adjusted count_](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#adjusted-count), which, in turn, is used to calculate span-based metrics. Mixing CP samplers with other types of samplers in most cases will lead to incorrect adjusted counts. The family of CP samplers has its own [composition rules](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#composition-rules), which correctly handle composing multiple CP samplers. + +## Prior art + +A number of composite samplers are already available as independent contributions +([RuleBasedRoutingSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/RuleBasedRoutingSampler.java), +[LinksBasedSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/LinksBasedSampler.java)). +Also, historically, some Span categorization was introduced by [JaegerRemoteSampler](https://www.jaegertracing.io/docs/1.54/sampling/#remote-sampling). + +This proposal aims at generalizing these ideas, and at providing a bit more formal specification for the behavior of the composite samplers. From 395bd0c57281f5efb457249a613ec3a741ef6006 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Wed, 28 Feb 2024 18:22:39 -0800 Subject: [PATCH 04/30] File rename --- text/{0000-Composite_Samplers.md => 0250-Composite_Samplers.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename text/{0000-Composite_Samplers.md => 0250-Composite_Samplers.md} (100%) diff --git a/text/0000-Composite_Samplers.md b/text/0250-Composite_Samplers.md similarity index 100% rename from text/0000-Composite_Samplers.md rename to text/0250-Composite_Samplers.md From fcb6a4b1ee34df6b06bb8acf856ac93460d31175 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Mon, 4 Mar 2024 16:30:37 -0800 Subject: [PATCH 05/30] Updated version, with examples --- text/0250-Composite_Samplers.md | 142 +++++++++++++++++++++++++------- 1 file changed, 114 insertions(+), 28 deletions(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index 2de373aa7..639a93744 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -1,7 +1,9 @@ -# Composite Samplers +# Composite Samplers Proposal This proposal addresses head-based sampling as described by the [Open Telemetry SDK](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#sampling). -It introduces additional composite samplers. Composite samplers use other samplers (delegates) to make sampling decisions. The composite samplers invoke the delegate samplers, but eventually make the final call. +It introduces additional _composite samplers_. Composite samplers use other samplers (delegates) to make sampling decisions. The composite samplers invoke the delegate samplers, but eventually make the final call. + +The new samplers proposed here are compatible with [Threshold propagation in trace state (OTEP 235)](https://github.com/open-telemetry/oteps/pull/235) as used by Consistent Probability samplers. Also see Draft PR 3910 [Probability Samplers based on W3C Trace Context Level 2](https://github.com/open-telemetry/opentelemetry-specification/pull/3910). ## Motivation @@ -13,11 +15,11 @@ The need for configuring head sampling has been explicitly or implicitly indicat ## The Goal -The goal of this proposal is to extend the set of standard SDK samplers by _Composite Samplers_ which make building advanced sampling configuration easier. +The goal of this proposal is to help creating advanced sampling configurations using pre-defined building blocks. Let's consider the following example of sampling requirements. It is believed that many users will have requirements following the same pattern. Most notable elements here are trace classification based on target URL, some spans requiring special handling, and putting a sanity cap on the total volume of exported spans. ### Example -Let's assume that a user wants to configure head sampling as follows: +Head-based sampling requirements. - for root spans: - drop all `/healthcheck` requests @@ -28,7 +30,7 @@ Let's assume that a user wants to configure head sampling as follows: - however, capture all calls to service `/foo` (even if the trace will be incomplete) - in any case, do not exceed 1000 spans/minute -It is hoped that the Composite Samplers presented here will be useful in constructing a sampling configuration satisfying the above case. +# New Samplers ## AnyOf @@ -42,9 +44,33 @@ Upon invocation of its `shouldSample` method, it MUST go through the whole list If any of the delegate Decisions is `RECORD_AND_SAMPLE`, the composite sampler MUST return `RECORD_AND_SAMPLE` Decision. Otherwise, if any of the delegate Decisions is `RECORD_ONLY`, the composite sampler MUST return `RECORD_ONLY` Decision. - The set of span Attributes to be added to the `Span` is the sum of the sets of Attributes as provided by the delegate samplers within their `SamplingResults`s. -- The `TraceState` to be used with the new `Span` is obtained by cumulatively applying all the potential modfications of the parent `TraceState` by the delegate samplers. +- The `TraceState` to be used with the new `Span` is obtained by cumulatively applying all the potential modfications of the parent `TraceState` by the delegate samplers, with special handling of the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry as described below. + +If the final sampling Decision is `DROP` or `RECORD_ONLY`, the `th` entry MUST be removed. +If the sampling Decision is `RECORD_AND_SAMPLE`, and there's no `th` entry in any of the `TraceState` provided by the delegates that decided to `RECORD_AND_SAMPLE`, the `th` entry MUST be also removed. +Otherwise, the resulting `TraceState` MUST contain `th` entry with the `THRESHOLD` value being the minimum of all the `THRESHOLD` values as reported by those delegates that decided to `RECORD_AND_SAMPLE`. + +Each delegate sampler MUST be given a chance to participate in the sampling decision as described above and MUST see the same _parent_ state. The order of the delegate samplers does not matter, as long as there's no overlap in the Attribute Keys or the trace state keys (other than `th`) that they use. -Each delegate sampler MUST be given a chance to participate in the sampling decision as described above and MUST see the same _parent_ state. The order of the delegate samplers does not matter, as long as there's no overlap in the Attribute Keys or the trace state keys they use. +## EachOf + +`EachOf` is a composite sampler which takes a non-empty list of Samplers (delegates) as the argument. + +Upon invocation of its `shouldSample` method, it MUST go through the whole list and invoke `shouldSample` method on each delegate sampler, passing the same arguments as received. + +`EachOf` sampler MUST return a `SamplingResult` which is constructed as follows: + +- The sampling Decision is based on the delegate sampling Decisions. If all of the delegate Decisions are `RECORD_AND_SAMPLE`, the composite sampler MUST return `RECORD_AND_SAMPLE` Decision as well. +If any of the delegate Decisions is `DROP`, the composite sampler MUST return `DROP` Decision. +Otherwise, if any of the delegate Decisions is `RECORD_ONLY`, the composite sampler MUST return `RECORD_ONLY` Decision. +- The set of span Attributes to be added to the `Span` is the sum of the sets of Attributes as provided by the delegate samplers within their `SamplingResults`s. +- The `TraceState` to be used with the new `Span` is obtained by cumulatively applying all the potential modfications of the parent `TraceState` by the delegate samplers, with special handling of the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry as described below. + +If the final sampling Decision is `DROP` or `RECORD_ONLY`, the `th` entry MUST be removed. +If the sampling Decision is `RECORD_AND_SAMPLE`, and there's no `th` entry in any of the `TraceState` provided by the delegates that decided to `RECORD_AND_SAMPLE`, the `th` entry MUST be also removed. +Otherwise, the resulting `TraceState` MUST contain `th` entry with the `THRESHOLD` value being the maximum of all the `THRESHOLD` values as reported by those delegates that decided to `RECORD_AND_SAMPLE`. + +Each delegate sampler MUST be given a chance to participate in the sampling decision as described above and MUST see the same _parent_ state. The order of the delegate samplers does not matter, as long as there's no overlap in the Attribute Keys or the trace state keys (other than `th`) that they use. ## Conjunction @@ -55,32 +81,36 @@ If the sampling Decision from the Second sampler is `RECORD_AND_SAMPLE`, the Con - The sampling Decision is `RECORD_AND_SAMPLE`. - The set of span Attributes to be added to the `Span` is the sum of the sets of Attributes as provided by the First and the Second sampler. -- The `TraceState` to be used with the new `Span` is obtained by cumulatively applying the potential modfications from the First and Second sampler. +- The `TraceState` to be used with the new `Span` is obtained by cumulatively applying the potential modfications from the First and Second sampler, with special handling of the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry as described below. + +If both First and Second samplers provided `th` entry in the returned `TraceState`, the resulting `TraceState` MUST contain `th` entry with the `THRESHOLD` being maximum of the `THRESHOLD`s provided by the First and Second samplers. Otherwise, the `th` entry MUST be removed. -If the sampling Decision from the Second sampler is `RECORD_ONLY` or `DROP`, the Conjunction sampler MUST retrun a `SamplingResult` which is constructed as follows: +If the sampling Decision from the Second sampler is `RECORD_ONLY` or `DROP`, the Conjunction sampler MUST return a `SamplingResult` which is constructed as follows: - The sampling Decision is `DROP`. - The set of span Attributes to be added to the `Span` is empty. -- The `TraceState` to be used with the new `Span` is the passed-in `TraceState`. +- The `TraceState` to be used with the new `Span` is the passed-in `TraceState`, but with the `th` entry removed. -## Rule based sampling +## RuleBased -For rule-based sampling (e.g. when sampling decision should depend on Span attributes), the Spans can be grouped into separate categories, and each category can use a different Sampler. Categorization of Spans is aided by `Predicates`. -The Predicates represent logical expressions which can access Span Attributes (or anything else available when the sampling decision is to be made), and perform tests on the accessible values. -For example, one can test if the target URL for a SERVER span matches a given pattern. +`RuleBased` is a composite sampler which performs `Span` categorization (e.g. when sampling decision depends on Span attributes) and sampling. +The Spans can be grouped into separate categories, and each category can use a different Sampler. +Categorization of Spans is aided by `Predicates`. ### Predicate -`Predicate` interface allows users to create custom categories based on information that is availabe at the time of making the sampling decision. +The Predicates represent logical expressions which can access Span Attributes (or anything else available when the sampling decision is to be made), and perform tests on the accessible values. +For example, one can test if the target URL for a SERVER span matches a given pattern. +`Predicate` interface allows users to create custom categories based on information that is available at the time of making the sampling decision. #### SpanMatches -This is a function/method for `Predicate`, which returns `true` if a given `Span` matches, i.e. belongs to the category described by the Predicate. +This is a routine/function/method for `Predicate`, which returns `true` if a given `Span` matches, i.e. belongs to the category described by the Predicate. -#### Required Arguments +#### Required Arguments for Predicates -The arguments represent the values that are made availabe for `ShouldSample`. +The arguments represent the values that are made available for `ShouldSample`. - `Context` with parent Span. - `TraceId` of the Span to be created. @@ -88,25 +118,81 @@ The arguments represent the values that are made availabe for `ShouldSample`. - Initial set of Attributes of the Span to be created. - Collection of links that will be associated with the Span to be created. -## RuleBased +### Required Arguments for RuleBased -`RuleBased` is a composite sampler which performs `Span` categorization and sampling. -It takes the following arguments: - `SpanKind` -- list of `Predicate`s -- list of `Sampler`s +- list of pairs (`Predicate`, `Sampler`) + +For making the sampling decision, if the `Span` kind matches the specified kind, the sampler goes through the list in the provided order and calls `SpanMatches` on `Predicate`s passing the `Span` as the argument. If a call returns `true`, the corresponding `Sampler` will be called to make the final sampling decision. If the `SpanKind` does not match, or none of the calls to `SpanMatches` yield `true`, the final decision is `DROP`. + +The order of `Predicate`s is essential. If more than one `Predicate` matches a `Span`, only the Sampler associated with the first matching `Predicate` will be used. + +# Summary + +## Example - sampling configuration 1 + +Going back to our example of sampling requirements, we can now configure the head sampler to support this particular case, using an informal notation of samplers and their arguments. +First, let's express the requirements for the ROOT spans as follows. + +``` +S1 = RuleBased(ROOT, { + (http.target == /healthcheck) => AlwaysOff, + (http.target == /checkout) => AlwaysOn, + true => TraceIdRatioBased(0.25) + }) +``` + +In the next step, we can build the sampler to handle non-root spans as well: -The `RuleBased` sampler SHOULD NOT accept lists of different length, i.e. it SHOULD report an error. Implementations MAY allow for replacing both lists with a list of pairs (`Predicate`, `Sampler`), if this is supported by the platform. +``` +S2 = ParentBased(S1) +``` -For making the sampling decision, if the `Span` kind matches the specified kind, the sampler goes through the lists in the declared order. If a Predicate matches the `Span` in question, the corresponding `Sampler` will be called to make the final sampling decision. If the `SpanKind` does not match, or none of the Predicates evaluates to `true`, the final decision is `DROP`. +The special case of calling service `/foo` can now be supported by: + +``` +S3 = AnyOf(S2, RuleBased(CLIENT, { (http.url == /foo) => AlwaysOn }) +``` + +Finally, the last step is to put a limit on the stream of exported spans. One of the available rate limiting sampler that we can use is Jaeger [RateLimitingSampler](https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/jaeger-remote-sampler/src/main/java/io/opentelemetry/sdk/extension/trace/jaeger/sampler/RateLimitingSampler.java): + +``` +S4 = Conjunction(S3, RateLimitingSampler(1000 * 60)) +``` +## Example - sampling configuration 2 + +Many users are interested in [Consistent Probability Sampling](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#consistent-probability-sampler) (CP), as it gives them a chance to calculate span-based metrics even when sampling is active. The configuration presented above uses the traditional samplers, which do not offer this benefit. + +Here is how an equivalent configuration can be put together using [CP samplers](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/tracestate-probability-sampling.md#samplers). In this example, the following implementations are used: + +- [ConsistentAlwaysOffSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/consistent-sampling/src/main/java/io/opentelemetry/contrib/sampler/consistent56/ConsistentAlwaysOffSampler.java) +- [ConsistentAlwaysOnSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/consistent-sampling/src/main/java/io/opentelemetry/contrib/sampler/consistent56/ConsistentAlwaysOnSampler.java) +- [ConsistentFixedThresholdSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/consistent-sampling/src/main/java/io/opentelemetry/contrib/sampler/consistent56/ConsistentFixedThresholdSampler.java) +- [ConsistentParentBasedSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/consistent-sampling/src/main/java/io/opentelemetry/contrib/sampler/consistent56/ConsistentParentBasedSampler.java) +- [ConsistentRateLimitingSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/consistent-sampling/src/main/java/io/opentelemetry/contrib/sampler/consistent56/ConsistentRateLimitingSampler.java) + +``` +S = EachOf( + AnyOf( + ConsistentParentBased( + RuleBased(ROOT, { + (http.target == /healthcheck) => ConsistentAlwaysOff, + (http.target == /checkout) => ConsistentAlwaysOn, + true => ConsistentFixedThreshold(0.25) + }), + RuleBased(CLIENT, { + (http.url == /foo) => ConsistentAlwaysOn + } + ), + ConsistentRateLimiting(1000 * 60) + ) +``` ## Limitations of composite samplers Not all samplers can participate as components of composite samplers without undesired or unexpected effects. Some samplers require that they _see_ each `Span` being created, even if the span is going to be dropped. Some samplers update the trace state or maintain internal state, and for their correct behavior it it is assumed that their sampling decisions will be honored by the tracer at the face value in all cases. A good example for this are rate limiting samplers which have to keep track of the rate of created spans and/or the rate of positive sampling decisions. -Placing such samplers as the first or second argument for the `Conjunction` sampler can sometimes preserve their correct behavior. - -A special attention is required for [consistent probability](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#consistent-probability-sampler) (CP) samplers. The sampling probability they record in trace-state is later used to calculate [_adjusted count_](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#adjusted-count), which, in turn, is used to calculate span-based metrics. Mixing CP samplers with other types of samplers in most cases will lead to incorrect adjusted counts. The family of CP samplers has its own [composition rules](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#composition-rules), which correctly handle composing multiple CP samplers. +A special attention is required for CP samplers. The sampling probability they record in trace-state is later used to calculate [_adjusted count_](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#adjusted-count), which, in turn, is used to calculate span-based metrics. While the composite samplers presented here are compatible with CP samplers, generally, mixing CP samplers with other types of samplers may lead to undefined or sometimes incorrect adjusted counts. ## Prior art From b37f05405077938152e39952a4cf965d4249b752 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Mon, 4 Mar 2024 16:42:19 -0800 Subject: [PATCH 06/30] Fixing markdown-lint errors --- text/0250-Composite_Samplers.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index 639a93744..dbcc5601e 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -159,6 +159,7 @@ Finally, the last step is to put a limit on the stream of exported spans. One of ``` S4 = Conjunction(S3, RateLimitingSampler(1000 * 60)) ``` + ## Example - sampling configuration 2 Many users are interested in [Consistent Probability Sampling](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#consistent-probability-sampler) (CP), as it gives them a chance to calculate span-based metrics even when sampling is active. The configuration presented above uses the traditional samplers, which do not offer this benefit. From 7bcd40b2ec85ef6ee5a078d65593f77b0b22694f Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Mon, 4 Mar 2024 16:51:09 -0800 Subject: [PATCH 07/30] Fixing more markdown-lint errors --- text/0250-Composite_Samplers.md | 33 ++++++++++++++++----------------- 1 file changed, 16 insertions(+), 17 deletions(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index dbcc5601e..dce3ae0e7 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -3,7 +3,7 @@ This proposal addresses head-based sampling as described by the [Open Telemetry SDK](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#sampling). It introduces additional _composite samplers_. Composite samplers use other samplers (delegates) to make sampling decisions. The composite samplers invoke the delegate samplers, but eventually make the final call. -The new samplers proposed here are compatible with [Threshold propagation in trace state (OTEP 235)](https://github.com/open-telemetry/oteps/pull/235) as used by Consistent Probability samplers. Also see Draft PR 3910 [Probability Samplers based on W3C Trace Context Level 2](https://github.com/open-telemetry/opentelemetry-specification/pull/3910). +The new samplers proposed here are compatible with [Threshold propagation in trace state (OTEP 235)](https://github.com/open-telemetry/oteps/pull/235) as used by Consistent Probability samplers. Also see Draft PR 3910 [Probability Samplers based on W3C Trace Context Level 2](https://github.com/open-telemetry/opentelemetry-specification/pull/3910). ## Motivation @@ -30,9 +30,9 @@ Head-based sampling requirements. - however, capture all calls to service `/foo` (even if the trace will be incomplete) - in any case, do not exceed 1000 spans/minute -# New Samplers +## New Samplers -## AnyOf +### AnyOf `AnyOf` is a composite sampler which takes a non-empty list of Samplers (delegates) as the argument. @@ -52,7 +52,7 @@ Otherwise, the resulting `TraceState` MUST contain `th` entry with the `THRESHOL Each delegate sampler MUST be given a chance to participate in the sampling decision as described above and MUST see the same _parent_ state. The order of the delegate samplers does not matter, as long as there's no overlap in the Attribute Keys or the trace state keys (other than `th`) that they use. -## EachOf +### EachOf `EachOf` is a composite sampler which takes a non-empty list of Samplers (delegates) as the argument. @@ -72,7 +72,7 @@ Otherwise, the resulting `TraceState` MUST contain `th` entry with the `THRESHOL Each delegate sampler MUST be given a chance to participate in the sampling decision as described above and MUST see the same _parent_ state. The order of the delegate samplers does not matter, as long as there's no overlap in the Attribute Keys or the trace state keys (other than `th`) that they use. -## Conjunction +### Conjunction `Conjunction` is a composite sampler which takes two Samplers (delegates) as the arguments. These delegate samplers will be hereby referenced as First and Second. @@ -83,7 +83,7 @@ If the sampling Decision from the Second sampler is `RECORD_AND_SAMPLE`, the Con - The set of span Attributes to be added to the `Span` is the sum of the sets of Attributes as provided by the First and the Second sampler. - The `TraceState` to be used with the new `Span` is obtained by cumulatively applying the potential modfications from the First and Second sampler, with special handling of the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry as described below. -If both First and Second samplers provided `th` entry in the returned `TraceState`, the resulting `TraceState` MUST contain `th` entry with the `THRESHOLD` being maximum of the `THRESHOLD`s provided by the First and Second samplers. Otherwise, the `th` entry MUST be removed. +If both First and Second samplers provided `th` entry in the returned `TraceState`, the resulting `TraceState` MUST contain `th` entry with the `THRESHOLD` being maximum of the `THRESHOLD`s provided by the First and Second samplers. Otherwise, the `th` entry MUST be removed. If the sampling Decision from the Second sampler is `RECORD_ONLY` or `DROP`, the Conjunction sampler MUST return a `SamplingResult` which is constructed as follows: @@ -91,24 +91,23 @@ If the sampling Decision from the Second sampler is `RECORD_ONLY` or `DROP`, the - The set of span Attributes to be added to the `Span` is empty. - The `TraceState` to be used with the new `Span` is the passed-in `TraceState`, but with the `th` entry removed. - -## RuleBased +### RuleBased `RuleBased` is a composite sampler which performs `Span` categorization (e.g. when sampling decision depends on Span attributes) and sampling. The Spans can be grouped into separate categories, and each category can use a different Sampler. Categorization of Spans is aided by `Predicates`. -### Predicate +#### Predicate The Predicates represent logical expressions which can access Span Attributes (or anything else available when the sampling decision is to be made), and perform tests on the accessible values. For example, one can test if the target URL for a SERVER span matches a given pattern. `Predicate` interface allows users to create custom categories based on information that is available at the time of making the sampling decision. -#### SpanMatches +##### SpanMatches This is a routine/function/method for `Predicate`, which returns `true` if a given `Span` matches, i.e. belongs to the category described by the Predicate. -#### Required Arguments for Predicates +##### Required Arguments for Predicates The arguments represent the values that are made available for `ShouldSample`. @@ -118,7 +117,7 @@ The arguments represent the values that are made available for `ShouldSample`. - Initial set of Attributes of the Span to be created. - Collection of links that will be associated with the Span to be created. -### Required Arguments for RuleBased +#### Required Arguments for RuleBased - `SpanKind` - list of pairs (`Predicate`, `Sampler`) @@ -127,9 +126,9 @@ For making the sampling decision, if the `Span` kind matches the specified kind, The order of `Predicate`s is essential. If more than one `Predicate` matches a `Span`, only the Sampler associated with the first matching `Predicate` will be used. -# Summary +## Summary -## Example - sampling configuration 1 +### Example - sampling configuration 1 Going back to our example of sampling requirements, we can now configure the head sampler to support this particular case, using an informal notation of samplers and their arguments. First, let's express the requirements for the ROOT spans as follows. @@ -160,7 +159,7 @@ Finally, the last step is to put a limit on the stream of exported spans. One of S4 = Conjunction(S3, RateLimitingSampler(1000 * 60)) ``` -## Example - sampling configuration 2 +### Example - sampling configuration 2 Many users are interested in [Consistent Probability Sampling](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#consistent-probability-sampler) (CP), as it gives them a chance to calculate span-based metrics even when sampling is active. The configuration presented above uses the traditional samplers, which do not offer this benefit. @@ -189,13 +188,13 @@ S = EachOf( ) ``` -## Limitations of composite samplers +### Limitations of composite samplers Not all samplers can participate as components of composite samplers without undesired or unexpected effects. Some samplers require that they _see_ each `Span` being created, even if the span is going to be dropped. Some samplers update the trace state or maintain internal state, and for their correct behavior it it is assumed that their sampling decisions will be honored by the tracer at the face value in all cases. A good example for this are rate limiting samplers which have to keep track of the rate of created spans and/or the rate of positive sampling decisions. A special attention is required for CP samplers. The sampling probability they record in trace-state is later used to calculate [_adjusted count_](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#adjusted-count), which, in turn, is used to calculate span-based metrics. While the composite samplers presented here are compatible with CP samplers, generally, mixing CP samplers with other types of samplers may lead to undefined or sometimes incorrect adjusted counts. -## Prior art +### Prior art A number of composite samplers are already available as independent contributions ([RuleBasedRoutingSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/RuleBasedRoutingSampler.java), From c72b06a4148d3e4d8364528013dd6d97b333c95a Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Tue, 12 Mar 2024 15:40:48 -0700 Subject: [PATCH 08/30] Modfied the behavior of Conjunction sampler. --- text/0250-Composite_Samplers.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index dce3ae0e7..eb02b81c3 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -76,20 +76,20 @@ Each delegate sampler MUST be given a chance to participate in the sampling deci `Conjunction` is a composite sampler which takes two Samplers (delegates) as the arguments. These delegate samplers will be hereby referenced as First and Second. -Upon invocation of its `shouldSample` method, the Conjunction sampler MUST invoke `shouldSample` method on the First sampler, passing the same arguments as received, and examine the received sampling Decision. Upon receiving `DROP` or `RECORD_ONLY` decision it MUST return the SamplingResult from the First sampler, and it MUST NOT proceed with querying the Second sampler. If the sampling decision from the First sampler is `RECORD_AND_SAMPLE`, the Conjunction sampler MUST invoke `shouldSample` method on the Second sampler. +Upon invocation of its `shouldSample` method, the Conjunction sampler MUST invoke `shouldSample` method on the First sampler, passing the same arguments as received, and examine the received sampling Decision. Upon receiving `DROP` or `RECORD_ONLY` decision it MUST return the SamplingResult from the First sampler, and it MUST NOT proceed with querying the Second sampler. If the sampling decision from the First sampler is `RECORD_AND_SAMPLE`, the Conjunction sampler MUST invoke `shouldSample` method on the Second sampler, effectively passing the `TraceState` received from the First sampler as the parent trace state. If the sampling Decision from the Second sampler is `RECORD_AND_SAMPLE`, the Conjunction sampler MUST return a `SamplingResult` which is constructed as follows: - The sampling Decision is `RECORD_AND_SAMPLE`. - The set of span Attributes to be added to the `Span` is the sum of the sets of Attributes as provided by the First and the Second sampler. - The `TraceState` to be used with the new `Span` is obtained by cumulatively applying the potential modfications from the First and Second sampler, with special handling of the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry as described below. -If both First and Second samplers provided `th` entry in the returned `TraceState`, the resulting `TraceState` MUST contain `th` entry with the `THRESHOLD` being maximum of the `THRESHOLD`s provided by the First and Second samplers. Otherwise, the `th` entry MUST be removed. +If both First and Second samplers provided `th` entry in the returned `TraceState`, and the value of the `THRESHOLD` from the First sampler is `0`, then the resulting `TraceState` MUST contain `th` entry with the `THRESHOLD`as provided by the Second sampler. Otherwise, the `th` entry MUST be removed. If the sampling Decision from the Second sampler is `RECORD_ONLY` or `DROP`, the Conjunction sampler MUST return a `SamplingResult` which is constructed as follows: - The sampling Decision is `DROP`. -- The set of span Attributes to be added to the `Span` is empty. -- The `TraceState` to be used with the new `Span` is the passed-in `TraceState`, but with the `th` entry removed. +- The set of span Attributes to be added to the `Span` is as provided by the First sampler. +- The `TraceState` to be used with the new `Span` is the `TraceState` provided by the First sampler, but with the `th` entry removed. ### RuleBased From 7cb67c1686bcf0ac170b4d31b85de86c7d2ca702 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Fri, 15 Mar 2024 08:47:33 -0700 Subject: [PATCH 09/30] Added description for Approach Two. --- text/0250-Composite_Samplers.md | 81 ++++++++++++++++++++++++++++++--- 1 file changed, 74 insertions(+), 7 deletions(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index eb02b81c3..385a5cbb7 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -3,7 +3,7 @@ This proposal addresses head-based sampling as described by the [Open Telemetry SDK](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#sampling). It introduces additional _composite samplers_. Composite samplers use other samplers (delegates) to make sampling decisions. The composite samplers invoke the delegate samplers, but eventually make the final call. -The new samplers proposed here are compatible with [Threshold propagation in trace state (OTEP 235)](https://github.com/open-telemetry/oteps/pull/235) as used by Consistent Probability samplers. Also see Draft PR 3910 [Probability Samplers based on W3C Trace Context Level 2](https://github.com/open-telemetry/opentelemetry-specification/pull/3910). +The new samplers proposed here are mostly compatible with Consistent Probability Samplers. For verbose description of this concept see [probability sampling](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/tracestate-probability-sampling.md). However, the technical details in that document are outdated. For the current proposal see [OTEP 235](https://github.com/open-telemetry/oteps/blob/main/text/trace/0235-sampling-threshold-in-trace-state.md). Also see Draft PR 3910 [Probability Samplers based on W3C Trace Context Level 2](https://github.com/open-telemetry/opentelemetry-specification/pull/3910). ## Motivation @@ -30,11 +30,17 @@ Head-based sampling requirements. - however, capture all calls to service `/foo` (even if the trace will be incomplete) - in any case, do not exceed 1000 spans/minute -## New Samplers +We present two quite different approaches to composite samplers. The first one uses only the current [sampling API](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#sampling). It can be applied to a large variety of samplers, but is not very efficient nor elegant. + +The second approach is applicable exclusively to Consistent Probability Samplers, but is more efficient and less prone to misconfiguration. + +## Approach One + +The following new composite samplers are proposed. ### AnyOf -`AnyOf` is a composite sampler which takes a non-empty list of Samplers (delegates) as the argument. +`AnyOf` is a composite sampler which takes a non-empty list of Samplers (delegates) as the argument. The intention is to make `RECORD_AND_SAMPLE` decision if __any of__ the delegates decides to `RECORD_AND_SAMPLE`. Upon invocation of its `shouldSample` method, it MUST go through the whole list and invoke `shouldSample` method on each delegate sampler, passing the same arguments as received. @@ -54,7 +60,7 @@ Each delegate sampler MUST be given a chance to participate in the sampling deci ### EachOf -`EachOf` is a composite sampler which takes a non-empty list of Samplers (delegates) as the argument. +`EachOf` is a composite sampler which takes a non-empty list of Samplers (delegates) as the argument. The intention is to make `RECORD_AND_SAMPLE` decision if __each of__ the delegates decides to `RECORD_AND_SAMPLE`. Upon invocation of its `shouldSample` method, it MUST go through the whole list and invoke `shouldSample` method on each delegate sampler, passing the same arguments as received. @@ -74,7 +80,7 @@ Each delegate sampler MUST be given a chance to participate in the sampling deci ### Conjunction -`Conjunction` is a composite sampler which takes two Samplers (delegates) as the arguments. These delegate samplers will be hereby referenced as First and Second. +`Conjunction` is a composite sampler which takes two Samplers (delegates) as the arguments. These delegate samplers will be hereby referenced as First and Second. This kind of composition forms conditional chaining of both samplers. Upon invocation of its `shouldSample` method, the Conjunction sampler MUST invoke `shouldSample` method on the First sampler, passing the same arguments as received, and examine the received sampling Decision. Upon receiving `DROP` or `RECORD_ONLY` decision it MUST return the SamplingResult from the First sampler, and it MUST NOT proceed with querying the Second sampler. If the sampling decision from the First sampler is `RECORD_AND_SAMPLE`, the Conjunction sampler MUST invoke `shouldSample` method on the Second sampler, effectively passing the `TraceState` received from the First sampler as the parent trace state. If the sampling Decision from the Second sampler is `RECORD_AND_SAMPLE`, the Conjunction sampler MUST return a `SamplingResult` which is constructed as follows: @@ -126,7 +132,7 @@ For making the sampling decision, if the `Span` kind matches the specified kind, The order of `Predicate`s is essential. If more than one `Predicate` matches a `Span`, only the Sampler associated with the first matching `Predicate` will be used. -## Summary +## Summary - Approach One ### Example - sampling configuration 1 @@ -141,6 +147,8 @@ S1 = RuleBased(ROOT, { }) ``` +Note: technically, `ROOT` is not a Span Kind, but is a special token matching all Spans with invalid parent context (i.e. the ROOT spans, regardless of their kind). + In the next step, we can build the sampler to handle non-root spans as well: ``` @@ -188,12 +196,71 @@ S = EachOf( ) ``` -### Limitations of composite samplers +### Limitations of composite samplers in approach One Not all samplers can participate as components of composite samplers without undesired or unexpected effects. Some samplers require that they _see_ each `Span` being created, even if the span is going to be dropped. Some samplers update the trace state or maintain internal state, and for their correct behavior it it is assumed that their sampling decisions will be honored by the tracer at the face value in all cases. A good example for this are rate limiting samplers which have to keep track of the rate of created spans and/or the rate of positive sampling decisions. A special attention is required for CP samplers. The sampling probability they record in trace-state is later used to calculate [_adjusted count_](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#adjusted-count), which, in turn, is used to calculate span-based metrics. While the composite samplers presented here are compatible with CP samplers, generally, mixing CP samplers with other types of samplers may lead to undefined or sometimes incorrect adjusted counts. +## Approach Two + +A principle of operation for Approach Two is that `ShouldSample` is invoked only once, on the root of the tree formed by composite samplers. All the logic provided by the composition of samplers is handled by calculating the threshold values, delegating the calculation downstream as necessary. + +### Consistent Probability Sampler API + +To make this approach possible, all Consistent Probability Samplers need to implement the following API, in addition to the standard Sampler API. This extension will be used by all composite samplers in Approach Two, as listed in the next sections. + +#### GetThreshold + +This is a routine/function/method for all Consistent Probability Samplers. Its purpose is to query the sampler to provide the rejection threshold value they would use had they been asked to make a sampling decision for a given span, however, without constructing the actual sampling Decision. + +#### Required Arguments for GetThreshold: + +The arguments are the same as for [`ShouldSample`](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#shouldsample). + +- `Context` with parent Span. +- `TraceId` of the Span to be created. +- Name of the Span to be created. +- `SpanKind` of the `Span` to be created. +- Initial set of Attributes of the Span to be created. +- Collection of links that will be associated with the Span to be created. + +#### Return value: + +The THRESHOLD value from range `0` to `2^56-1` (a 56-bit unsigned integer number) if the sampler is going to make a probability based sampling decision. Values outside of this range can be used for other situations (such as AllwaysOff decisions), or to eventually support sampling decisions other than `DROP` or `RECORD_AND_SAMPLE`. + +### ConsistentRuleBased + +This composite sampler re-uses the concept of Predicates from Approach One. + +#### Required Arguments for ConsistentRuleBased + +- `SpanKind` +- list of pairs (`Predicate`, `ConsistentProbabilitySampler`) + +For calculating the rejection THRESHOLD, if the `Span` kind matches the specified kind, the sampler goes through the list in the provided order and calls `SpanMatches` on `Predicate`s passing the `Span` as the argument. If a call returns `true`, the result is as returned by `GetThreshold` called on the corresponding `ConsistentProbabilitySampler`. If the `SpanKind` does not match, or none of the calls to `SpanMatches` yield `true`, the result is obtained by calling `GetThreshold` on `ConsistentAlwaysOffSampler`. + +### ConsistentAnyOf + +`ConsistentAnyOf` is a composite sampler which takes a non-empty list of ConsistentProbabilitySamplers (delegates) as the argument. The intention is to make a positive sampling decision if __any of__ the delegates would make a positive decision. + +Upon invocation of its `GetThreshold` method, it MUST go through the whole list and invoke `GetTheshold` method on each delegate sampler, passing the same arguments as received. + +`ConsistentAnyOf` sampler MUST return a THRESHOLD which is constructed as follows: + +- If any of the delegates returned a threshold value from the range of `0` to `2^56-1`, the resulting threshold is the minimum value from the set of results from within that range. +- Otherwise, the result is obtained by calling `GetThreshold` on `ConsistentAlwaysOffSampler`. + +Each delegate sampler MUST be given a chance to participate in calculating the threshold as described above and MUST see the same argument values. The order of the delegate samplers does not matter. + +### ConsistentRateLimiting + +_Description pending. For now, look at the explanations given in comments for this OTEP._ + +### Limitations of composite samplers in Approach Two + +While making sampling decisions with samplers from Approach Two is more efficient and avoids dealing with non-mainstream cases, it puts some limits on the capabilities of the Consistent Probability Samplers. In particular, a custom CP sampler that wishes to add a span `Attribute` or modify TraceState will be out of luck if it is used as a delegate. + ### Prior art A number of composite samplers are already available as independent contributions From ab8e9627055177dbf4c80ee84e3ce5119a20a9a3 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Wed, 20 Mar 2024 17:53:12 -0700 Subject: [PATCH 10/30] Added description of ConsistentRateLimited, another example, and a few clarifying sentences. --- text/0250-Composite_Samplers.md | 46 ++++++++++++++++++++++++++++----- 1 file changed, 40 insertions(+), 6 deletions(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index 385a5cbb7..de08628fa 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -128,7 +128,7 @@ The arguments represent the values that are made available for `ShouldSample`. - `SpanKind` - list of pairs (`Predicate`, `Sampler`) -For making the sampling decision, if the `Span` kind matches the specified kind, the sampler goes through the list in the provided order and calls `SpanMatches` on `Predicate`s passing the `Span` as the argument. If a call returns `true`, the corresponding `Sampler` will be called to make the final sampling decision. If the `SpanKind` does not match, or none of the calls to `SpanMatches` yield `true`, the final decision is `DROP`. +For making the sampling decision, if the `Span` kind matches the specified kind, the sampler goes through the list in the provided order and calls `SpanMatches` on `Predicate`s passing the same arguments as received by `ShouldSample`. If a call returns `true`, the corresponding `Sampler` will be called to make the final sampling decision. If the `SpanKind` does not match, or none of the calls to `SpanMatches` yield `true`, the final decision is `DROP`. The order of `Predicate`s is essential. If more than one `Predicate` matches a `Span`, only the Sampler associated with the first matching `Predicate` will be used. @@ -227,7 +227,7 @@ The arguments are the same as for [`ShouldSample`](https://github.com/open-telem #### Return value: -The THRESHOLD value from range `0` to `2^56-1` (a 56-bit unsigned integer number) if the sampler is going to make a probability based sampling decision. Values outside of this range can be used for other situations (such as AllwaysOff decisions), or to eventually support sampling decisions other than `DROP` or `RECORD_AND_SAMPLE`. +The THRESHOLD value from range `0` to `2^56-1` (a 56-bit unsigned integer number) if the sampler is ready to make a probability based sampling decision. Values outside of this range can be used for other situations (such as AllwaysOff decisions), or to eventually support equivalents of sampling decisions other than `DROP` or `RECORD_AND_SAMPLE`. ### ConsistentRuleBased @@ -238,13 +238,13 @@ This composite sampler re-uses the concept of Predicates from Approach One. - `SpanKind` - list of pairs (`Predicate`, `ConsistentProbabilitySampler`) -For calculating the rejection THRESHOLD, if the `Span` kind matches the specified kind, the sampler goes through the list in the provided order and calls `SpanMatches` on `Predicate`s passing the `Span` as the argument. If a call returns `true`, the result is as returned by `GetThreshold` called on the corresponding `ConsistentProbabilitySampler`. If the `SpanKind` does not match, or none of the calls to `SpanMatches` yield `true`, the result is obtained by calling `GetThreshold` on `ConsistentAlwaysOffSampler`. +For calculating the rejection THRESHOLD, if the `Span` kind matches the specified kind, the sampler goes through the list in the provided order and calls `SpanMatches` on `Predicate`s passing the same arguments as received. If a call returns `true`, the result is as returned by `GetThreshold` called on the corresponding `ConsistentProbabilitySampler`. If the `SpanKind` does not match, or none of the calls to `SpanMatches` yield `true`, the result is obtained by calling `GetThreshold` on `ConsistentAlwaysOffSampler`. ### ConsistentAnyOf `ConsistentAnyOf` is a composite sampler which takes a non-empty list of ConsistentProbabilitySamplers (delegates) as the argument. The intention is to make a positive sampling decision if __any of__ the delegates would make a positive decision. -Upon invocation of its `GetThreshold` method, it MUST go through the whole list and invoke `GetTheshold` method on each delegate sampler, passing the same arguments as received. +Upon invocation of its `GetThreshold` function, it MUST go through the whole list and invoke `GetTheshold` function on each delegate sampler, passing the same arguments as received. `ConsistentAnyOf` sampler MUST return a THRESHOLD which is constructed as follows: @@ -255,13 +255,47 @@ Each delegate sampler MUST be given a chance to participate in calculating the t ### ConsistentRateLimiting -_Description pending. For now, look at the explanations given in comments for this OTEP._ +`ConsistentRateLimiting` is a composite sampler that helps control the average rate of sampled spans while allowing another sampler (the delegate) to provide sampling hints. + +#### Required Arguments for ConsistentRateLimiting + +- ConsistentProbabilitySampler (delegate) +- maximum sampling (throughput) target rate + +The sampler SHOULD measure and keep the average rate of incoming spans, and therefore also of the desired ratio between the incoming span rate to the target span rate. +Upon invocation of its `GetThreshold` function, the composite sampler MUST get the threshold from the delegate sampler, passing the same arguments as received. +If using the obtained threshold value as the final threshold would entail sampling more spans than the declared target rate, the sampler SHOULD increase the threshold to a value that would meet the target rate. Several algorithms can be used for threshold adjustment, no particular behavior is prescried by the specification though. + +When using `ConsistentRateLimiting` in our requirements example as a replacement for `EachOf` and `RateLimiting`, we are left with no use case for any direct equivalent to `EachOf` in Approach Two. + +## Summary - Approach Two + +### Example - sampling configuration with Approach Two + +With the samplers introduced by Approach Two, our example requirements can be coded in a very similar way as with ApproachOne. However, the work of the samplers configured this way forms a tree of `GetThreshold` invocations rather than `ShouldSample` invocations as in ApproachOne. + +``` +S = ConsistentRateLimiting( + ConsistentAnyOf( + ConsistentParentBased( + ConsistentRuleBased(ROOT, { + (http.target == /healthcheck) => ConsistentAlwaysOff, + (http.target == /checkout) => ConsistentAlwaysOn, + true => ConsistentFixedThreshold(0.25) + }), + ConsistentRuleBased(CLIENT, { + (http.url == /foo) => ConsistentAlwaysOn + } + ), + 1000 * 60 + ) +``` ### Limitations of composite samplers in Approach Two While making sampling decisions with samplers from Approach Two is more efficient and avoids dealing with non-mainstream cases, it puts some limits on the capabilities of the Consistent Probability Samplers. In particular, a custom CP sampler that wishes to add a span `Attribute` or modify TraceState will be out of luck if it is used as a delegate. -### Prior art +## Prior art A number of composite samplers are already available as independent contributions ([RuleBasedRoutingSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/RuleBasedRoutingSampler.java), From 47084f2df50bc39d0300917311de8b554c17379b Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Mon, 8 Apr 2024 18:30:13 -0700 Subject: [PATCH 11/30] Changing description of sampler as suggested --- text/0250-Composite_Samplers.md | 47 ++++++++++++++++++--------------- 1 file changed, 25 insertions(+), 22 deletions(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index de08628fa..9d114cd03 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -42,60 +42,63 @@ The following new composite samplers are proposed. `AnyOf` is a composite sampler which takes a non-empty list of Samplers (delegates) as the argument. The intention is to make `RECORD_AND_SAMPLE` decision if __any of__ the delegates decides to `RECORD_AND_SAMPLE`. -Upon invocation of its `shouldSample` method, it MUST go through the whole list and invoke `shouldSample` method on each delegate sampler, passing the same arguments as received. +Upon invocation of its `shouldSample` method, it MUST go through the whole list and invoke `shouldSample` method on each delegate sampler, passing the same arguments as received, and collecting the delegates' sampling Decisions. -`AnyOf` sampler MUST return a `SamplingResult` which is constructed as follows: +`AnyOf` sampler MUST return a `SamplingResult` with the following elements. -- The sampling Decision is based on the delegate sampling Decisions. If all of the delegate Decisions are `DROP`, the composite sampler MUST return `DROP` Decision as well. +- If all of the delegate Decisions are `DROP`, the composite sampler MUST return `DROP` Decision as well. If any of the delegate Decisions is `RECORD_AND_SAMPLE`, the composite sampler MUST return `RECORD_AND_SAMPLE` Decision. Otherwise, if any of the delegate Decisions is `RECORD_ONLY`, the composite sampler MUST return `RECORD_ONLY` Decision. -- The set of span Attributes to be added to the `Span` is the sum of the sets of Attributes as provided by the delegate samplers within their `SamplingResults`s. -- The `TraceState` to be used with the new `Span` is obtained by cumulatively applying all the potential modfications of the parent `TraceState` by the delegate samplers, with special handling of the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry as described below. +- The set of span Attributes to be added to the `Span` is the sum of the sets of Attributes as provided by those delegate samplers which produced a sampling Decision other than `DROP`. In case of conflicting attribute keys, the attribute definition from the last delegate that uses that key takes effect. +- The `TraceState` to be used with the new `Span` is obtained by cumulatively applying all the potential modfications of the parent `TraceState` by the delegate samplers. In case of conflicting entry keys, the entry definition provided by the last delegate that uses that key takes effect. However, the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry gets special handling as described below. If the final sampling Decision is `DROP` or `RECORD_ONLY`, the `th` entry MUST be removed. If the sampling Decision is `RECORD_AND_SAMPLE`, and there's no `th` entry in any of the `TraceState` provided by the delegates that decided to `RECORD_AND_SAMPLE`, the `th` entry MUST be also removed. Otherwise, the resulting `TraceState` MUST contain `th` entry with the `THRESHOLD` value being the minimum of all the `THRESHOLD` values as reported by those delegates that decided to `RECORD_AND_SAMPLE`. -Each delegate sampler MUST be given a chance to participate in the sampling decision as described above and MUST see the same _parent_ state. The order of the delegate samplers does not matter, as long as there's no overlap in the Attribute Keys or the trace state keys (other than `th`) that they use. +Each delegate sampler MUST be given a chance to participate in the sampling decision as described above and MUST see the same _parent_ state. The resulting sampling Decision does not depend on the order of the delegate samplers. ### EachOf `EachOf` is a composite sampler which takes a non-empty list of Samplers (delegates) as the argument. The intention is to make `RECORD_AND_SAMPLE` decision if __each of__ the delegates decides to `RECORD_AND_SAMPLE`. -Upon invocation of its `shouldSample` method, it MUST go through the whole list and invoke `shouldSample` method on each delegate sampler, passing the same arguments as received. +Upon invocation of its `shouldSample` method, it MUST go through the whole list and invoke `shouldSample` method on each delegate sampler, passing the same arguments as received, and collecting the delegates' sampling Decisions. -`EachOf` sampler MUST return a `SamplingResult` which is constructed as follows: +`EachOf` sampler MUST return a `SamplingResult` with the following elements. -- The sampling Decision is based on the delegate sampling Decisions. If all of the delegate Decisions are `RECORD_AND_SAMPLE`, the composite sampler MUST return `RECORD_AND_SAMPLE` Decision as well. +- If all of the delegate Decisions are `RECORD_AND_SAMPLE`, the composite sampler MUST return `RECORD_AND_SAMPLE` Decision as well. If any of the delegate Decisions is `DROP`, the composite sampler MUST return `DROP` Decision. Otherwise, if any of the delegate Decisions is `RECORD_ONLY`, the composite sampler MUST return `RECORD_ONLY` Decision. -- The set of span Attributes to be added to the `Span` is the sum of the sets of Attributes as provided by the delegate samplers within their `SamplingResults`s. -- The `TraceState` to be used with the new `Span` is obtained by cumulatively applying all the potential modfications of the parent `TraceState` by the delegate samplers, with special handling of the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry as described below. +- If the resulting sampling Decision is `DROP`, the set of span Attributes to be added to the `Span` is empty. Otherwise, it is the sum of the sets of Attributes as provided by the delegate samplers within their `SamplingResults`s. In case of conflicting attribute keys, the attribute definition from the last delegate that uses that key takes effect. +- The `TraceState` to be used with the new `Span` is obtained by cumulatively applying all the potential modfications of the parent `TraceState` by the delegate samplers. In case of conflicting entry keys, the entry definition provided by the last delegate that uses that key takes effect. However, the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry gets special handling as described below. If the final sampling Decision is `DROP` or `RECORD_ONLY`, the `th` entry MUST be removed. -If the sampling Decision is `RECORD_AND_SAMPLE`, and there's no `th` entry in any of the `TraceState` provided by the delegates that decided to `RECORD_AND_SAMPLE`, the `th` entry MUST be also removed. -Otherwise, the resulting `TraceState` MUST contain `th` entry with the `THRESHOLD` value being the maximum of all the `THRESHOLD` values as reported by those delegates that decided to `RECORD_AND_SAMPLE`. +If the sampling Decision is `RECORD_AND_SAMPLE`, and there's no `th` entry in any of the `TraceState` provided by the delegates, the `th` entry MUST be also removed. +Otherwise, the resulting `TraceState` MUST contain `th` entry with the `THRESHOLD` value being the maximum of all the `THRESHOLD` values as reported the delegates. -Each delegate sampler MUST be given a chance to participate in the sampling decision as described above and MUST see the same _parent_ state. The order of the delegate samplers does not matter, as long as there's no overlap in the Attribute Keys or the trace state keys (other than `th`) that they use. +Each delegate sampler MUST be given a chance to participate in the sampling decision as described above and MUST see the same _parent_ state. The resulting sampling Decision does not depend on the order of the delegate samplers. ### Conjunction `Conjunction` is a composite sampler which takes two Samplers (delegates) as the arguments. These delegate samplers will be hereby referenced as First and Second. This kind of composition forms conditional chaining of both samplers. Upon invocation of its `shouldSample` method, the Conjunction sampler MUST invoke `shouldSample` method on the First sampler, passing the same arguments as received, and examine the received sampling Decision. Upon receiving `DROP` or `RECORD_ONLY` decision it MUST return the SamplingResult from the First sampler, and it MUST NOT proceed with querying the Second sampler. If the sampling decision from the First sampler is `RECORD_AND_SAMPLE`, the Conjunction sampler MUST invoke `shouldSample` method on the Second sampler, effectively passing the `TraceState` received from the First sampler as the parent trace state. + If the sampling Decision from the Second sampler is `RECORD_AND_SAMPLE`, the Conjunction sampler MUST return a `SamplingResult` which is constructed as follows: - The sampling Decision is `RECORD_AND_SAMPLE`. -- The set of span Attributes to be added to the `Span` is the sum of the sets of Attributes as provided by the First and the Second sampler. -- The `TraceState` to be used with the new `Span` is obtained by cumulatively applying the potential modfications from the First and Second sampler, with special handling of the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry as described below. +- The set of span Attributes to be added to the `Span` is the sum of the sets of Attributes as provided by both samplers. +- The `TraceState` to be used with the new `Span` is as provided by the Second sampler, with special handling of the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry. +If the First sampler did not provide `th` entry in the returned `TraceState`, or if the value of the corresponding `THRESHOLD` is not `0`, then the `th` entry MUST be removed from the resulting `TraceState`. -If both First and Second samplers provided `th` entry in the returned `TraceState`, and the value of the `THRESHOLD` from the First sampler is `0`, then the resulting `TraceState` MUST contain `th` entry with the `THRESHOLD`as provided by the Second sampler. Otherwise, the `th` entry MUST be removed. - -If the sampling Decision from the Second sampler is `RECORD_ONLY` or `DROP`, the Conjunction sampler MUST return a `SamplingResult` which is constructed as follows: +If the sampling Decision from the Second sampler is `DROP` or `RECORD_ONLY`, the Conjunction sampler MUST return a `SamplingResult` which is constructed as follows: - The sampling Decision is `DROP`. -- The set of span Attributes to be added to the `Span` is as provided by the First sampler. -- The `TraceState` to be used with the new `Span` is the `TraceState` provided by the First sampler, but with the `th` entry removed. +- The set of span Attributes to be added to the `Span` is empty. +- The `TraceState` to be used with the new `Span` is the `TraceState` provided by the Second sampler, but with the `th` entry removed. + +The `Conjunction` sampler can be useful in a special case where the user wants to keep a group of traces, for example belonging to an end user session, together - meaning to make the same sampling decisions for all traces belonging to the group, as much as possible. +One way of achieving this behavior for consistent probability samplers is to give all traces belonging to the group the same _randomness_ (represented by `r-value`), based on some criteria shared by all traces belonging to the group. This can be done by a special sampler which would provide the required `r-value` for all `ROOT` spans of the involved traces. When using such a sampler as the First delegate for `Conjunction`, this functionality can be encapsulated in a separate sampler, without making any changes to the current SDK specification. ### RuleBased @@ -196,7 +199,7 @@ S = EachOf( ) ``` -### Limitations of composite samplers in approach One +### Limitations of composite samplers in Approach One Not all samplers can participate as components of composite samplers without undesired or unexpected effects. Some samplers require that they _see_ each `Span` being created, even if the span is going to be dropped. Some samplers update the trace state or maintain internal state, and for their correct behavior it it is assumed that their sampling decisions will be honored by the tracer at the face value in all cases. A good example for this are rate limiting samplers which have to keep track of the rate of created spans and/or the rate of positive sampling decisions. From a6d370545bc4e9c5681c066fc5a3707a3a076f6b Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Fri, 19 Apr 2024 10:34:10 -0700 Subject: [PATCH 12/30] Update text/0250-Composite_Samplers.md Co-authored-by: Otmar Ertl --- text/0250-Composite_Samplers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index 9d114cd03..d1bd7e243 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -69,7 +69,7 @@ Upon invocation of its `shouldSample` method, it MUST go through the whole list - If all of the delegate Decisions are `RECORD_AND_SAMPLE`, the composite sampler MUST return `RECORD_AND_SAMPLE` Decision as well. If any of the delegate Decisions is `DROP`, the composite sampler MUST return `DROP` Decision. Otherwise, if any of the delegate Decisions is `RECORD_ONLY`, the composite sampler MUST return `RECORD_ONLY` Decision. -- If the resulting sampling Decision is `DROP`, the set of span Attributes to be added to the `Span` is empty. Otherwise, it is the sum of the sets of Attributes as provided by the delegate samplers within their `SamplingResults`s. In case of conflicting attribute keys, the attribute definition from the last delegate that uses that key takes effect. +- If the resulting sampling Decision is `DROP`, the set of span Attributes to be added to the `Span` is empty. Otherwise, it is the sum of the sets of Attributes as provided by the delegate samplers within their `SamplingResult`s. In case of conflicting attribute keys, the attribute definition from the last delegate that uses that key takes effect. - The `TraceState` to be used with the new `Span` is obtained by cumulatively applying all the potential modfications of the parent `TraceState` by the delegate samplers. In case of conflicting entry keys, the entry definition provided by the last delegate that uses that key takes effect. However, the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry gets special handling as described below. If the final sampling Decision is `DROP` or `RECORD_ONLY`, the `th` entry MUST be removed. From cae3b743597c7ee8bc15eee36137fc8dd3a72bc1 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Fri, 19 Apr 2024 10:36:53 -0700 Subject: [PATCH 13/30] Update text/0250-Composite_Samplers.md Co-authored-by: Otmar Ertl --- text/0250-Composite_Samplers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index d1bd7e243..03b3c04db 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -50,7 +50,7 @@ Upon invocation of its `shouldSample` method, it MUST go through the whole list If any of the delegate Decisions is `RECORD_AND_SAMPLE`, the composite sampler MUST return `RECORD_AND_SAMPLE` Decision. Otherwise, if any of the delegate Decisions is `RECORD_ONLY`, the composite sampler MUST return `RECORD_ONLY` Decision. - The set of span Attributes to be added to the `Span` is the sum of the sets of Attributes as provided by those delegate samplers which produced a sampling Decision other than `DROP`. In case of conflicting attribute keys, the attribute definition from the last delegate that uses that key takes effect. -- The `TraceState` to be used with the new `Span` is obtained by cumulatively applying all the potential modfications of the parent `TraceState` by the delegate samplers. In case of conflicting entry keys, the entry definition provided by the last delegate that uses that key takes effect. However, the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry gets special handling as described below. +- The `TraceState` to be used with the new `Span` is obtained by cumulatively applying all the potential modifications of the parent `TraceState` by the delegate samplers. In case of conflicting entry keys, the entry definition provided by the last delegate that uses that key takes effect. However, the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry gets special handling as described below. If the final sampling Decision is `DROP` or `RECORD_ONLY`, the `th` entry MUST be removed. If the sampling Decision is `RECORD_AND_SAMPLE`, and there's no `th` entry in any of the `TraceState` provided by the delegates that decided to `RECORD_AND_SAMPLE`, the `th` entry MUST be also removed. From 0d3ce189ab3b24f50e693f4a559cf2df734ea6a7 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Fri, 19 Apr 2024 11:25:23 -0700 Subject: [PATCH 14/30] Update text/0250-Composite_Samplers.md Co-authored-by: Otmar Ertl --- text/0250-Composite_Samplers.md | 1 - 1 file changed, 1 deletion(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index 03b3c04db..d64c3dfdc 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -222,7 +222,6 @@ This is a routine/function/method for all Consistent Probability Samplers. Its p The arguments are the same as for [`ShouldSample`](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#shouldsample). - `Context` with parent Span. -- `TraceId` of the Span to be created. - Name of the Span to be created. - `SpanKind` of the `Span` to be created. - Initial set of Attributes of the Span to be created. From 9a7afb33ea7f6ab06963b76211b1d8ab06f21d2d Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Mon, 22 Apr 2024 09:00:25 -0700 Subject: [PATCH 15/30] Update text/0250-Composite_Samplers.md Co-authored-by: Otmar Ertl --- text/0250-Composite_Samplers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index d64c3dfdc..d7e935cb7 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -87,7 +87,7 @@ Upon invocation of its `shouldSample` method, the Conjunction sampler MUST invok If the sampling Decision from the Second sampler is `RECORD_AND_SAMPLE`, the Conjunction sampler MUST return a `SamplingResult` which is constructed as follows: - The sampling Decision is `RECORD_AND_SAMPLE`. -- The set of span Attributes to be added to the `Span` is the sum of the sets of Attributes as provided by both samplers. +- The set of span Attributes to be added to the `Span` is the union of the sets of Attributes as provided by both samplers. - The `TraceState` to be used with the new `Span` is as provided by the Second sampler, with special handling of the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry. If the First sampler did not provide `th` entry in the returned `TraceState`, or if the value of the corresponding `THRESHOLD` is not `0`, then the `th` entry MUST be removed from the resulting `TraceState`. From 129c99c2132d9c835d44b82cb69231aee9238204 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Mon, 22 Apr 2024 09:09:53 -0700 Subject: [PATCH 16/30] Update text/0250-Composite_Samplers.md Co-authored-by: Otmar Ertl --- text/0250-Composite_Samplers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index d7e935cb7..687fbf3da 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -219,7 +219,7 @@ This is a routine/function/method for all Consistent Probability Samplers. Its p #### Required Arguments for GetThreshold: -The arguments are the same as for [`ShouldSample`](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#shouldsample). +The arguments are the same as for [`ShouldSample`](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#shouldsample) except for the `TraceId`. - `Context` with parent Span. - Name of the Span to be created. From 10ba0c1a1f36c6ba929299a4b815c4e065a21b34 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Wed, 1 May 2024 11:48:25 -0700 Subject: [PATCH 17/30] Describing a two-phase processing for Approach Two. --- text/0250-Composite_Samplers.md | 79 +++++++++++++++++++++------------ 1 file changed, 51 insertions(+), 28 deletions(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index 687fbf3da..796189eb8 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -1,7 +1,7 @@ # Composite Samplers Proposal This proposal addresses head-based sampling as described by the [Open Telemetry SDK](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#sampling). -It introduces additional _composite samplers_. Composite samplers use other samplers (delegates) to make sampling decisions. The composite samplers invoke the delegate samplers, but eventually make the final call. +It introduces additional _composite samplers_. Composite samplers use other samplers (_delegates_ or _children_) to make sampling decisions. The composite samplers invoke the delegate samplers, but eventually make the final call. The new samplers proposed here are mostly compatible with Consistent Probability Samplers. For verbose description of this concept see [probability sampling](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/tracestate-probability-sampling.md). However, the technical details in that document are outdated. For the current proposal see [OTEP 235](https://github.com/open-telemetry/oteps/blob/main/text/trace/0235-sampling-threshold-in-trace-state.md). Also see Draft PR 3910 [Probability Samplers based on W3C Trace Context Level 2](https://github.com/open-telemetry/opentelemetry-specification/pull/3910). @@ -31,9 +31,10 @@ Head-based sampling requirements. - in any case, do not exceed 1000 spans/minute We present two quite different approaches to composite samplers. The first one uses only the current [sampling API](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#sampling). It can be applied to a large variety of samplers, but is not very efficient nor elegant. - The second approach is applicable exclusively to Consistent Probability Samplers, but is more efficient and less prone to misconfiguration. +__Note__: both approaches call for calculating unions of Attribute sets. Whenever such union is constructed, in case of conflicting attribute keys, the attribute definition from the last set that uses that key takes effect. Similarly, whenever modifications of `Tracestate` are chained, in case of conflicting keys, the last modification erases the previous values. + ## Approach One The following new composite samplers are proposed. @@ -49,12 +50,12 @@ Upon invocation of its `shouldSample` method, it MUST go through the whole list - If all of the delegate Decisions are `DROP`, the composite sampler MUST return `DROP` Decision as well. If any of the delegate Decisions is `RECORD_AND_SAMPLE`, the composite sampler MUST return `RECORD_AND_SAMPLE` Decision. Otherwise, if any of the delegate Decisions is `RECORD_ONLY`, the composite sampler MUST return `RECORD_ONLY` Decision. -- The set of span Attributes to be added to the `Span` is the sum of the sets of Attributes as provided by those delegate samplers which produced a sampling Decision other than `DROP`. In case of conflicting attribute keys, the attribute definition from the last delegate that uses that key takes effect. -- The `TraceState` to be used with the new `Span` is obtained by cumulatively applying all the potential modifications of the parent `TraceState` by the delegate samplers. In case of conflicting entry keys, the entry definition provided by the last delegate that uses that key takes effect. However, the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry gets special handling as described below. +- The set of span Attributes to be added to the `Span` is the union of the sets of Attributes as provided by those delegate samplers which produced a sampling Decision other than `DROP`. +- The `Tracestate` to be used with the new `Span` is obtained by cumulatively applying all the potential modifications of the parent `Tracestate` by the delegate samplers. However, the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry gets special handling as described below. If the final sampling Decision is `DROP` or `RECORD_ONLY`, the `th` entry MUST be removed. -If the sampling Decision is `RECORD_AND_SAMPLE`, and there's no `th` entry in any of the `TraceState` provided by the delegates that decided to `RECORD_AND_SAMPLE`, the `th` entry MUST be also removed. -Otherwise, the resulting `TraceState` MUST contain `th` entry with the `THRESHOLD` value being the minimum of all the `THRESHOLD` values as reported by those delegates that decided to `RECORD_AND_SAMPLE`. +If the sampling Decision is `RECORD_AND_SAMPLE`, and there's no `th` entry in any of the `Tracestate` provided by the delegates that decided to `RECORD_AND_SAMPLE`, the `th` entry MUST be also removed. +Otherwise, the resulting `Tracestate` MUST contain `th` entry with the `THRESHOLD` value being the minimum of all the `THRESHOLD` values as reported by those delegates that decided to `RECORD_AND_SAMPLE`. Each delegate sampler MUST be given a chance to participate in the sampling decision as described above and MUST see the same _parent_ state. The resulting sampling Decision does not depend on the order of the delegate samplers. @@ -69,12 +70,12 @@ Upon invocation of its `shouldSample` method, it MUST go through the whole list - If all of the delegate Decisions are `RECORD_AND_SAMPLE`, the composite sampler MUST return `RECORD_AND_SAMPLE` Decision as well. If any of the delegate Decisions is `DROP`, the composite sampler MUST return `DROP` Decision. Otherwise, if any of the delegate Decisions is `RECORD_ONLY`, the composite sampler MUST return `RECORD_ONLY` Decision. -- If the resulting sampling Decision is `DROP`, the set of span Attributes to be added to the `Span` is empty. Otherwise, it is the sum of the sets of Attributes as provided by the delegate samplers within their `SamplingResult`s. In case of conflicting attribute keys, the attribute definition from the last delegate that uses that key takes effect. -- The `TraceState` to be used with the new `Span` is obtained by cumulatively applying all the potential modfications of the parent `TraceState` by the delegate samplers. In case of conflicting entry keys, the entry definition provided by the last delegate that uses that key takes effect. However, the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry gets special handling as described below. +- If the resulting sampling Decision is `DROP`, the set of span Attributes to be added to the `Span` is empty. Otherwise, it is the union of the sets of Attributes as provided by the delegate samplers within their `SamplingResult`s. +- The `Tracestate` to be used with the new `Span` is obtained by cumulatively applying all the potential modfications of the parent `Tracestate` by the delegate samplers. However, the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry gets special handling as described below. If the final sampling Decision is `DROP` or `RECORD_ONLY`, the `th` entry MUST be removed. -If the sampling Decision is `RECORD_AND_SAMPLE`, and there's no `th` entry in any of the `TraceState` provided by the delegates, the `th` entry MUST be also removed. -Otherwise, the resulting `TraceState` MUST contain `th` entry with the `THRESHOLD` value being the maximum of all the `THRESHOLD` values as reported the delegates. +If the sampling Decision is `RECORD_AND_SAMPLE`, and there's no `th` entry in any of the `Tracestate` provided by the delegates, the `th` entry MUST be also removed. +Otherwise, the resulting `Tracestate` MUST contain `th` entry with the `THRESHOLD` value being the maximum of all the `THRESHOLD` values as reported the delegates. Each delegate sampler MUST be given a chance to participate in the sampling decision as described above and MUST see the same _parent_ state. The resulting sampling Decision does not depend on the order of the delegate samplers. @@ -82,20 +83,20 @@ Each delegate sampler MUST be given a chance to participate in the sampling deci `Conjunction` is a composite sampler which takes two Samplers (delegates) as the arguments. These delegate samplers will be hereby referenced as First and Second. This kind of composition forms conditional chaining of both samplers. -Upon invocation of its `shouldSample` method, the Conjunction sampler MUST invoke `shouldSample` method on the First sampler, passing the same arguments as received, and examine the received sampling Decision. Upon receiving `DROP` or `RECORD_ONLY` decision it MUST return the SamplingResult from the First sampler, and it MUST NOT proceed with querying the Second sampler. If the sampling decision from the First sampler is `RECORD_AND_SAMPLE`, the Conjunction sampler MUST invoke `shouldSample` method on the Second sampler, effectively passing the `TraceState` received from the First sampler as the parent trace state. +Upon invocation of its `shouldSample` method, the Conjunction sampler MUST invoke `shouldSample` method on the First sampler, passing the same arguments as received, and examine the received sampling Decision. Upon receiving `DROP` or `RECORD_ONLY` decision it MUST return the SamplingResult from the First sampler, and it MUST NOT proceed with querying the Second sampler. If the sampling decision from the First sampler is `RECORD_AND_SAMPLE`, the Conjunction sampler MUST invoke `shouldSample` method on the Second sampler, effectively passing the `Tracestate` received from the First sampler as the parent trace state. If the sampling Decision from the Second sampler is `RECORD_AND_SAMPLE`, the Conjunction sampler MUST return a `SamplingResult` which is constructed as follows: - The sampling Decision is `RECORD_AND_SAMPLE`. - The set of span Attributes to be added to the `Span` is the union of the sets of Attributes as provided by both samplers. -- The `TraceState` to be used with the new `Span` is as provided by the Second sampler, with special handling of the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry. -If the First sampler did not provide `th` entry in the returned `TraceState`, or if the value of the corresponding `THRESHOLD` is not `0`, then the `th` entry MUST be removed from the resulting `TraceState`. +- The `Tracestate` to be used with the new `Span` is as provided by the Second sampler, with special handling of the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry. +If the First sampler did not provide `th` entry in the returned `Tracestate`, or if the value of the corresponding `THRESHOLD` is not `0`, then the `th` entry MUST be removed from the resulting `Tracestate`. If the sampling Decision from the Second sampler is `DROP` or `RECORD_ONLY`, the Conjunction sampler MUST return a `SamplingResult` which is constructed as follows: - The sampling Decision is `DROP`. - The set of span Attributes to be added to the `Span` is empty. -- The `TraceState` to be used with the new `Span` is the `TraceState` provided by the Second sampler, but with the `th` entry removed. +- The `Tracestate` to be used with the new `Span` is the `Tracestate` provided by the Second sampler, but with the `th` entry removed. The `Conjunction` sampler can be useful in a special case where the user wants to keep a group of traces, for example belonging to an end user session, together - meaning to make the same sampling decisions for all traces belonging to the group, as much as possible. One way of achieving this behavior for consistent probability samplers is to give all traces belonging to the group the same _randomness_ (represented by `r-value`), based on some criteria shared by all traces belonging to the group. This can be done by a special sampler which would provide the required `r-value` for all `ROOT` spans of the involved traces. When using such a sampler as the First delegate for `Conjunction`, this functionality can be encapsulated in a separate sampler, without making any changes to the current SDK specification. @@ -213,11 +214,11 @@ A principle of operation for Approach Two is that `ShouldSample` is invoked only To make this approach possible, all Consistent Probability Samplers need to implement the following API, in addition to the standard Sampler API. This extension will be used by all composite samplers in Approach Two, as listed in the next sections. -#### GetThreshold +#### GetSamplingAdvice -This is a routine/function/method for all Consistent Probability Samplers. Its purpose is to query the sampler to provide the rejection threshold value they would use had they been asked to make a sampling decision for a given span, however, without constructing the actual sampling Decision. +This is a routine/function/method for all Consistent Probability Samplers. Its purpose is to query the sampler about the activities it would perform had it been asked to make a sampling decision for a given span, however, without constructing the actual sampling Decision. -#### Required Arguments for GetThreshold: +#### Required Arguments for GetSamplingAdvice: The arguments are the same as for [`ShouldSample`](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#shouldsample) except for the `TraceId`. @@ -229,7 +230,21 @@ The arguments are the same as for [`ShouldSample`](https://github.com/open-telem #### Return value: -The THRESHOLD value from range `0` to `2^56-1` (a 56-bit unsigned integer number) if the sampler is ready to make a probability based sampling decision. Values outside of this range can be used for other situations (such as AllwaysOff decisions), or to eventually support equivalents of sampling decisions other than `DROP` or `RECORD_AND_SAMPLE`. +The return value is a structure (`SamplingAdvice`) with the following elements: + +- The THRESHOLD value represented as a 14-character hexadecimal string, with value of `null` representing non-probabilistic `DROP` decision, +- A function (`GetAttributes`) that provides a set of Attributes to be added to the Span in case of positive final sampling decision, +- A function (`GetTraceState`) that given an input `Tracestate` and sampling Decision provides a `Tracestate` to be associated with the Span. The samplers SHOULD NOT add or modify the `th` value for the `ot` key within these functions. + +#### Constructing `SamplingResult` + +The process of constructing the final `SamplingResult` in response to a call to `ShouldSample` on the root sampler of the composite samplers tree consists of the following steps. + +- The sampler gets its own `SamplingAdvice`, it is a recursive process as described below (unless the sampler is a leaf), +- The sampler compares the received THRESHOLD value with the trace Randomness value to arrive at the final sampling `Decision`, +- In case of a positive sampling decision the sampler calls the received `GetAttributes` function to determine the set of Attributes to be added to the Span, in most cases it will be a recursive step, +- The sampler calls the received `GetTraceState` function passing the parent `Tracestate` and the final sampling `Decision` to get the new `Tracestate` to be associated with the Span - again, in most cases this is a recursive step, +- The sampler modfies (or removes) the `th` value for the `ot` key in the `Tracestate` according to the final sampling `Decision` and the THRESHOLD used in the second step above. ### ConsistentRuleBased @@ -240,20 +255,21 @@ This composite sampler re-uses the concept of Predicates from Approach One. - `SpanKind` - list of pairs (`Predicate`, `ConsistentProbabilitySampler`) -For calculating the rejection THRESHOLD, if the `Span` kind matches the specified kind, the sampler goes through the list in the provided order and calls `SpanMatches` on `Predicate`s passing the same arguments as received. If a call returns `true`, the result is as returned by `GetThreshold` called on the corresponding `ConsistentProbabilitySampler`. If the `SpanKind` does not match, or none of the calls to `SpanMatches` yield `true`, the result is obtained by calling `GetThreshold` on `ConsistentAlwaysOffSampler`. +For calculating the `SamplingAdvice`, if the `Span` kind matches the specified kind, the sampler goes through the list in the provided order and calls `SpanMatches` on `Predicate`s passing the same arguments as received. If a call returns `true`, the result is as returned by `GetSamplingAdvice` called on the corresponding `ConsistentProbabilitySampler`. If the `SpanKind` does not match, or none of the calls to `SpanMatches` yield `true`, the result is obtained by calling `GetSamplingAdvice` on `ConsistentAlwaysOffSampler`. ### ConsistentAnyOf `ConsistentAnyOf` is a composite sampler which takes a non-empty list of ConsistentProbabilitySamplers (delegates) as the argument. The intention is to make a positive sampling decision if __any of__ the delegates would make a positive decision. -Upon invocation of its `GetThreshold` function, it MUST go through the whole list and invoke `GetTheshold` function on each delegate sampler, passing the same arguments as received. +Upon invocation of its `GetSamplingAdvice` function, it MUST go through the whole list and invoke `GetSamplingAdvice` function on each delegate sampler, passing the same arguments as received. -`ConsistentAnyOf` sampler MUST return a THRESHOLD which is constructed as follows: +`ConsistentAnyOf` sampler MUST return a `SamplingAdvice` which is constructed as follows: -- If any of the delegates returned a threshold value from the range of `0` to `2^56-1`, the resulting threshold is the minimum value from the set of results from within that range. -- Otherwise, the result is obtained by calling `GetThreshold` on `ConsistentAlwaysOffSampler`. +- If any of the delegates returned a non-`null` threshold value, the resulting threshold is the lexicographical minimum value from the set of those non-`null` values, otherwise `null`. +- The `GetAttributes` function calculates the union of Attribute sets as returned by the calls to `GetAttributes` function for each delegate (in the declared order) that returned a non-`null` threshold value, with later Attributes clobbering older Attributes with the same key. +- The `GetTraceState` function makes a chain of calls to the `GetTraceState` functions as returned by the delegates, passing the received `Tracestate` as argument to subsequent calls and returning the last value received. -Each delegate sampler MUST be given a chance to participate in calculating the threshold as described above and MUST see the same argument values. The order of the delegate samplers does not matter. +Each delegate sampler MUST be given a chance to participate in calculating the `SamplingAdvice` as described above and MUST see the same argument values. The order of the delegate samplers does not affect the final sampling `Decision`. ### ConsistentRateLimiting @@ -265,8 +281,13 @@ Each delegate sampler MUST be given a chance to participate in calculating the t - maximum sampling (throughput) target rate The sampler SHOULD measure and keep the average rate of incoming spans, and therefore also of the desired ratio between the incoming span rate to the target span rate. -Upon invocation of its `GetThreshold` function, the composite sampler MUST get the threshold from the delegate sampler, passing the same arguments as received. -If using the obtained threshold value as the final threshold would entail sampling more spans than the declared target rate, the sampler SHOULD increase the threshold to a value that would meet the target rate. Several algorithms can be used for threshold adjustment, no particular behavior is prescried by the specification though. +Upon invocation of its `GetSamplingAdvice` function, the composite sampler MUST get the `SamplingAdvice` from the delegate sampler, passing the same arguments as received. + +The returned `SamplingAdvice` is constructed as follows. + +- If using the obtained threshold value as the final threshold would entail sampling more spans than the declared target rate, the sampler SHOULD increase the threshold to a value that would meet the target rate. Several algorithms can be used for threshold adjustment, no particular behavior is prescried by the specification though. +- The `GetAttributes` function returns the union of the set of Attributes returned by calling the delegate's `GetAttributes` and own Attributes. +- The `GetTraceState` function returns the `Tracestate` as returned by calling `GetTraceState` from the delegate's `SamplingAdvice`. When using `ConsistentRateLimiting` in our requirements example as a replacement for `EachOf` and `RateLimiting`, we are left with no use case for any direct equivalent to `EachOf` in Approach Two. @@ -274,7 +295,7 @@ When using `ConsistentRateLimiting` in our requirements example as a replacement ### Example - sampling configuration with Approach Two -With the samplers introduced by Approach Two, our example requirements can be coded in a very similar way as with ApproachOne. However, the work of the samplers configured this way forms a tree of `GetThreshold` invocations rather than `ShouldSample` invocations as in ApproachOne. +With the samplers introduced by Approach Two, our example requirements can be coded in a very similar way as with Approach One. However, the work of the samplers configured this way forms a tree of `GetSamplingAdvice` invocations rather than `ShouldSample` invocations as in Approach One. ``` S = ConsistentRateLimiting( @@ -295,7 +316,9 @@ S = ConsistentRateLimiting( ### Limitations of composite samplers in Approach Two -While making sampling decisions with samplers from Approach Two is more efficient and avoids dealing with non-mainstream cases, it puts some limits on the capabilities of the Consistent Probability Samplers. In particular, a custom CP sampler that wishes to add a span `Attribute` or modify TraceState will be out of luck if it is used as a delegate. +Making sampling decisions with samplers from Approach Two is more efficient than in Approach One, especially if, platform permitting, `null` values can be used for `GetAttributes` and `GetTraceState` functions to represent the prevailing trivial cases of _no-new-attributes_ and _no-special-trace-state-keys_. The only limitation is that it operates exclusively within the domain of Consistent Probability Sampling (not necessarilly a bad thing). + +Developers of Consistent Probability Samplers should consider that the samplers they provide might be used in composite samplers, and that that the sampling Decision they advise might be different from the final sampling Decision. ## Prior art From 1c15c2b4eb28506c4b3ed4b9ce70fd23c5d0686a Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Wed, 1 May 2024 11:51:49 -0700 Subject: [PATCH 18/30] Fixing a typo --- text/0250-Composite_Samplers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index 796189eb8..ed140ee5a 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -316,7 +316,7 @@ S = ConsistentRateLimiting( ### Limitations of composite samplers in Approach Two -Making sampling decisions with samplers from Approach Two is more efficient than in Approach One, especially if, platform permitting, `null` values can be used for `GetAttributes` and `GetTraceState` functions to represent the prevailing trivial cases of _no-new-attributes_ and _no-special-trace-state-keys_. The only limitation is that it operates exclusively within the domain of Consistent Probability Sampling (not necessarilly a bad thing). +Making sampling decisions with samplers from Approach Two is more efficient than in Approach One, especially if, platform permitting, `null` values can be used for `GetAttributes` and `GetTraceState` functions to represent the prevailing trivial cases of _no-new-attributes_ and _no-special-trace-state-keys_. The only limitation is that it operates exclusively within the domain of Consistent Probability Sampling (not necessarily a bad thing). Developers of Consistent Probability Samplers should consider that the samplers they provide might be used in composite samplers, and that that the sampling Decision they advise might be different from the final sampling Decision. From a0bfb1bb87d454d6084a214ff80029875374f58b Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Fri, 3 May 2024 10:17:53 -0700 Subject: [PATCH 19/30] Minor adjustments and corrections, introducing Composable Sampler. --- text/0250-Composite_Samplers.md | 39 +++++++++++++++++++-------------- 1 file changed, 23 insertions(+), 16 deletions(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index ed140ee5a..67f4dcdd4 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -3,7 +3,7 @@ This proposal addresses head-based sampling as described by the [Open Telemetry SDK](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#sampling). It introduces additional _composite samplers_. Composite samplers use other samplers (_delegates_ or _children_) to make sampling decisions. The composite samplers invoke the delegate samplers, but eventually make the final call. -The new samplers proposed here are mostly compatible with Consistent Probability Samplers. For verbose description of this concept see [probability sampling](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/tracestate-probability-sampling.md). However, the technical details in that document are outdated. For the current proposal see [OTEP 235](https://github.com/open-telemetry/oteps/blob/main/text/trace/0235-sampling-threshold-in-trace-state.md). Also see Draft PR 3910 [Probability Samplers based on W3C Trace Context Level 2](https://github.com/open-telemetry/opentelemetry-specification/pull/3910). +The new samplers proposed here are mostly compatible with Consistent Probability Samplers. For detailed description of this concept see [probability sampling](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/tracestate-probability-sampling.md). However, the technical details in that document are outdated. For the current proposal see [OTEP 235](https://github.com/open-telemetry/oteps/blob/main/text/trace/0235-sampling-threshold-in-trace-state.md). Also see Draft PR 3910 [Probability Samplers based on W3C Trace Context Level 2](https://github.com/open-telemetry/opentelemetry-specification/pull/3910). ## Motivation @@ -13,9 +13,11 @@ The need for configuring head sampling has been explicitly or implicitly indicat - issue [1060](https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/1060): Exclude URLs from Tracing - issue [1844](https://github.com/open-telemetry/opentelemetry-specification/issues/1844): Composite Sampler +Unfortunately, some of the valuable ideas flowing at the sampling SIG meetings never got recorded at the time of their inception, see the comments under [OTEP 240: A Sampling Configuration proposal](https://github.com/open-telemetry/oteps/pull/240) for some examples. + ## The Goal -The goal of this proposal is to help creating advanced sampling configurations using pre-defined building blocks. Let's consider the following example of sampling requirements. It is believed that many users will have requirements following the same pattern. Most notable elements here are trace classification based on target URL, some spans requiring special handling, and putting a sanity cap on the total volume of exported spans. +The goal of this proposal is to help creating advanced sampling configurations using pre-defined building blocks. Let's consider the following example of sampling requirements. It is believed that many users will have requirements following a similar pattern. Most notable elements here are trace classification based on target URL, some spans requiring special handling, and putting a sanity cap on the total volume of exported spans. ### Example @@ -31,7 +33,8 @@ Head-based sampling requirements. - in any case, do not exceed 1000 spans/minute We present two quite different approaches to composite samplers. The first one uses only the current [sampling API](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#sampling). It can be applied to a large variety of samplers, but is not very efficient nor elegant. -The second approach is applicable exclusively to Consistent Probability Samplers, but is more efficient and less prone to misconfiguration. + +The second approach is applicable exclusively to Consistent Probability Samplers, but is more efficient and less prone to misconfiguration. It requires additional API to be provided by the delegate samplers. __Note__: both approaches call for calculating unions of Attribute sets. Whenever such union is constructed, in case of conflicting attribute keys, the attribute definition from the last set that uses that key takes effect. Similarly, whenever modifications of `Tracestate` are chained, in case of conflicting keys, the last modification erases the previous values. @@ -206,17 +209,20 @@ Not all samplers can participate as components of composite samplers without und A special attention is required for CP samplers. The sampling probability they record in trace-state is later used to calculate [_adjusted count_](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#adjusted-count), which, in turn, is used to calculate span-based metrics. While the composite samplers presented here are compatible with CP samplers, generally, mixing CP samplers with other types of samplers may lead to undefined or sometimes incorrect adjusted counts. +The need to encode and decode the `Tracestate` multiple times affects performance of the composite samplers. This drawback is eliminated in Approach Two. + ## Approach Two A principle of operation for Approach Two is that `ShouldSample` is invoked only once, on the root of the tree formed by composite samplers. All the logic provided by the composition of samplers is handled by calculating the threshold values, delegating the calculation downstream as necessary. -### Consistent Probability Sampler API +### New API -To make this approach possible, all Consistent Probability Samplers need to implement the following API, in addition to the standard Sampler API. This extension will be used by all composite samplers in Approach Two, as listed in the next sections. +To make this approach possible, all Consistent Probability Samplers which participate in the samplers composition need to implement the following API, in addition to the standard Sampler API. We will use the term _Composable Sampler_ to dentote Consistent Probability Samplers which provide the new API and conform to the rules described here. +The composite samplers in Approach Two are Composable Samplers as well. #### GetSamplingAdvice -This is a routine/function/method for all Consistent Probability Samplers. Its purpose is to query the sampler about the activities it would perform had it been asked to make a sampling decision for a given span, however, without constructing the actual sampling Decision. +This is a routine/function/method for all Composable Samplers. Its purpose is to query the sampler about the activities it would perform had it been asked to make a sampling decision for a given span, however, without constructing the actual sampling Decision. #### Required Arguments for GetSamplingAdvice: @@ -234,7 +240,7 @@ The return value is a structure (`SamplingAdvice`) with the following elements: - The THRESHOLD value represented as a 14-character hexadecimal string, with value of `null` representing non-probabilistic `DROP` decision, - A function (`GetAttributes`) that provides a set of Attributes to be added to the Span in case of positive final sampling decision, -- A function (`GetTraceState`) that given an input `Tracestate` and sampling Decision provides a `Tracestate` to be associated with the Span. The samplers SHOULD NOT add or modify the `th` value for the `ot` key within these functions. +- A function (`UpdateTraceState`) that given an input `Tracestate` and sampling Decision provides a `Tracestate` to be associated with the Span. The samplers SHOULD NOT add or modify the `th` value for the `ot` key within these functions. #### Constructing `SamplingResult` @@ -243,7 +249,7 @@ The process of constructing the final `SamplingResult` in response to a call to - The sampler gets its own `SamplingAdvice`, it is a recursive process as described below (unless the sampler is a leaf), - The sampler compares the received THRESHOLD value with the trace Randomness value to arrive at the final sampling `Decision`, - In case of a positive sampling decision the sampler calls the received `GetAttributes` function to determine the set of Attributes to be added to the Span, in most cases it will be a recursive step, -- The sampler calls the received `GetTraceState` function passing the parent `Tracestate` and the final sampling `Decision` to get the new `Tracestate` to be associated with the Span - again, in most cases this is a recursive step, +- The sampler calls the received `UpdateTraceState` function passing the parent `Tracestate` and the final sampling `Decision` to get the new `Tracestate` to be associated with the Span - again, in most cases this is a recursive step, - The sampler modfies (or removes) the `th` value for the `ot` key in the `Tracestate` according to the final sampling `Decision` and the THRESHOLD used in the second step above. ### ConsistentRuleBased @@ -253,13 +259,13 @@ This composite sampler re-uses the concept of Predicates from Approach One. #### Required Arguments for ConsistentRuleBased - `SpanKind` -- list of pairs (`Predicate`, `ConsistentProbabilitySampler`) +- list of pairs (`Predicate`, `ComposableSampler`) -For calculating the `SamplingAdvice`, if the `Span` kind matches the specified kind, the sampler goes through the list in the provided order and calls `SpanMatches` on `Predicate`s passing the same arguments as received. If a call returns `true`, the result is as returned by `GetSamplingAdvice` called on the corresponding `ConsistentProbabilitySampler`. If the `SpanKind` does not match, or none of the calls to `SpanMatches` yield `true`, the result is obtained by calling `GetSamplingAdvice` on `ConsistentAlwaysOffSampler`. +For calculating the `SamplingAdvice`, if the `Span` kind matches the specified kind, the sampler goes through the list in the provided order and calls `SpanMatches` on `Predicate`s passing the same arguments as received. If a call returns `true`, the result is as returned by `GetSamplingAdvice` called on the corresponding `ComposableSampler`. If the `SpanKind` does not match, or none of the calls to `SpanMatches` yield `true`, the result is obtained by calling `GetSamplingAdvice` on `ConsistentAlwaysOffSampler`. ### ConsistentAnyOf -`ConsistentAnyOf` is a composite sampler which takes a non-empty list of ConsistentProbabilitySamplers (delegates) as the argument. The intention is to make a positive sampling decision if __any of__ the delegates would make a positive decision. +`ConsistentAnyOf` is a composite sampler which takes a non-empty list of ComposableSamplers (delegates) as the argument. The intention is to make a positive sampling decision if __any of__ the delegates would make a positive decision. Upon invocation of its `GetSamplingAdvice` function, it MUST go through the whole list and invoke `GetSamplingAdvice` function on each delegate sampler, passing the same arguments as received. @@ -267,7 +273,7 @@ Upon invocation of its `GetSamplingAdvice` function, it MUST go through the whol - If any of the delegates returned a non-`null` threshold value, the resulting threshold is the lexicographical minimum value from the set of those non-`null` values, otherwise `null`. - The `GetAttributes` function calculates the union of Attribute sets as returned by the calls to `GetAttributes` function for each delegate (in the declared order) that returned a non-`null` threshold value, with later Attributes clobbering older Attributes with the same key. -- The `GetTraceState` function makes a chain of calls to the `GetTraceState` functions as returned by the delegates, passing the received `Tracestate` as argument to subsequent calls and returning the last value received. +- The `UpdateTraceState` function makes a chain of calls to the `UpdateTraceState` functions as returned by the delegates, passing the received `Tracestate` as argument to subsequent calls and returning the last value received. Each delegate sampler MUST be given a chance to participate in calculating the `SamplingAdvice` as described above and MUST see the same argument values. The order of the delegate samplers does not affect the final sampling `Decision`. @@ -277,7 +283,7 @@ Each delegate sampler MUST be given a chance to participate in calculating the ` #### Required Arguments for ConsistentRateLimiting -- ConsistentProbabilitySampler (delegate) +- ComposableSampler (delegate) - maximum sampling (throughput) target rate The sampler SHOULD measure and keep the average rate of incoming spans, and therefore also of the desired ratio between the incoming span rate to the target span rate. @@ -287,9 +293,10 @@ The returned `SamplingAdvice` is constructed as follows. - If using the obtained threshold value as the final threshold would entail sampling more spans than the declared target rate, the sampler SHOULD increase the threshold to a value that would meet the target rate. Several algorithms can be used for threshold adjustment, no particular behavior is prescried by the specification though. - The `GetAttributes` function returns the union of the set of Attributes returned by calling the delegate's `GetAttributes` and own Attributes. -- The `GetTraceState` function returns the `Tracestate` as returned by calling `GetTraceState` from the delegate's `SamplingAdvice`. +- The `UpdateTraceState` function returns the `Tracestate` as returned by calling `UpdateTraceState` from the delegate's `SamplingAdvice`. When using `ConsistentRateLimiting` in our requirements example as a replacement for `EachOf` and `RateLimiting`, we are left with no use case for any direct equivalent to `EachOf` in Approach Two. +However, if such a need arises, `ConsistentEachOf` can be easily defined. ## Summary - Approach Two @@ -316,9 +323,9 @@ S = ConsistentRateLimiting( ### Limitations of composite samplers in Approach Two -Making sampling decisions with samplers from Approach Two is more efficient than in Approach One, especially if, platform permitting, `null` values can be used for `GetAttributes` and `GetTraceState` functions to represent the prevailing trivial cases of _no-new-attributes_ and _no-special-trace-state-keys_. The only limitation is that it operates exclusively within the domain of Consistent Probability Sampling (not necessarily a bad thing). +Making sampling decisions with samplers from Approach Two is more efficient than in Approach One, especially if, platform permitting, `null` values can be used for `GetAttributes` and `UpdateTraceState` functions to represent the prevailing trivial cases of _no-new-attributes_ and _no-special-trace-state-keys_. The only limitation of this approach that it operates exclusively within the domain of Composable Samplers (a subset of Consistent Probability Samplers). -Developers of Consistent Probability Samplers should consider that the samplers they provide might be used in composite samplers, and that that the sampling Decision they advise might be different from the final sampling Decision. +Developers of Composable Samplers should consider that the sampling Decision they advise might be different from the final sampling Decision. ## Prior art From e5447947ae1c08d240b80d9c7162fa3ee662a9c2 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Fri, 3 May 2024 11:09:23 -0700 Subject: [PATCH 20/30] Adding more references --- text/0250-Composite_Samplers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index 67f4dcdd4..9b7c99a55 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -13,7 +13,7 @@ The need for configuring head sampling has been explicitly or implicitly indicat - issue [1060](https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/1060): Exclude URLs from Tracing - issue [1844](https://github.com/open-telemetry/opentelemetry-specification/issues/1844): Composite Sampler -Unfortunately, some of the valuable ideas flowing at the sampling SIG meetings never got recorded at the time of their inception, see the comments under [OTEP 240: A Sampling Configuration proposal](https://github.com/open-telemetry/oteps/pull/240) for some examples. +Unfortunately, some of the valuable ideas flowing at the sampling SIG meetings never got recorded at the time of their inception, but see [Sampling SIG Research Notes](https://github.com/open-telemetry/oteps/pull/213) or the comments under [OTEP 240: A Sampling Configuration proposal](https://github.com/open-telemetry/oteps/pull/240) for some examples. ## The Goal From 635b21608c350b5ca282a23485c850a07199f6a9 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Thu, 6 Jun 2024 15:17:02 -0700 Subject: [PATCH 21/30] Update text/0250-Composite_Samplers.md Co-authored-by: J. Kalyana Sundaram --- text/0250-Composite_Samplers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index 9b7c99a55..e41e6f962 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -7,7 +7,7 @@ The new samplers proposed here are mostly compatible with Consistent Probability ## Motivation -The need for configuring head sampling has been explicitly or implicitly indicated in several discussions, both within the [Samplig SIG](https://docs.google.com/document/d/1gASMhmxNt9qCa8czEMheGlUW2xpORiYoD7dBD7aNtbQ) and in the wider community. Some of the discussions are going back a number of years, see for example +The need for configuring head sampling has been explicitly or implicitly indicated in several discussions, both within the [Sampling SIG](https://docs.google.com/document/d/1gASMhmxNt9qCa8czEMheGlUW2xpORiYoD7dBD7aNtbQ) and in the wider community. Some of the discussions are going back a number of years, see for example - issue [173](https://github.com/open-telemetry/opentelemetry-specification/issues/173): Way to ignore healthcheck traces when using automatic tracer across all languages? - issue [1060](https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/1060): Exclude URLs from Tracing From 06cc32d45f9aeba892f161bd84bb7e41000da85f Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Thu, 6 Jun 2024 15:17:18 -0700 Subject: [PATCH 22/30] Update text/0250-Composite_Samplers.md Co-authored-by: J. Kalyana Sundaram --- text/0250-Composite_Samplers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index e41e6f962..91a883cb3 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -217,7 +217,7 @@ A principle of operation for Approach Two is that `ShouldSample` is invoked only ### New API -To make this approach possible, all Consistent Probability Samplers which participate in the samplers composition need to implement the following API, in addition to the standard Sampler API. We will use the term _Composable Sampler_ to dentote Consistent Probability Samplers which provide the new API and conform to the rules described here. +To make this approach possible, all Consistent Probability Samplers which participate in the samplers composition need to implement the following API, in addition to the standard Sampler API. We will use the term _Composable Sampler_ to denote Consistent Probability Samplers which provide the new API and conform to the rules described here. The composite samplers in Approach Two are Composable Samplers as well. #### GetSamplingAdvice From 5cc2174c6a798a2ac1eb00fc199f7b308252fc5b Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Thu, 6 Jun 2024 15:17:42 -0700 Subject: [PATCH 23/30] Update text/0250-Composite_Samplers.md Co-authored-by: J. Kalyana Sundaram --- text/0250-Composite_Samplers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index 91a883cb3..3d63bc7ff 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -250,7 +250,7 @@ The process of constructing the final `SamplingResult` in response to a call to - The sampler compares the received THRESHOLD value with the trace Randomness value to arrive at the final sampling `Decision`, - In case of a positive sampling decision the sampler calls the received `GetAttributes` function to determine the set of Attributes to be added to the Span, in most cases it will be a recursive step, - The sampler calls the received `UpdateTraceState` function passing the parent `Tracestate` and the final sampling `Decision` to get the new `Tracestate` to be associated with the Span - again, in most cases this is a recursive step, -- The sampler modfies (or removes) the `th` value for the `ot` key in the `Tracestate` according to the final sampling `Decision` and the THRESHOLD used in the second step above. +- The sampler modifies (or removes) the `th` value for the `ot` key in the `Tracestate` according to the final sampling `Decision` and the THRESHOLD used in the second step above. ### ConsistentRuleBased From d3c82a32a5e451bd62e1f590fa8017a320e351e8 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Tue, 11 Jun 2024 10:14:02 -0700 Subject: [PATCH 24/30] Changing SamplingAdvice to SamplingIntent. Adding clarifications and cleaning up inconsistencies. --- text/0250-Composite_Samplers.md | 100 ++++++++++++++++++-------------- 1 file changed, 58 insertions(+), 42 deletions(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index 3d63bc7ff..95f579931 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -1,13 +1,17 @@ # Composite Samplers Proposal This proposal addresses head-based sampling as described by the [Open Telemetry SDK](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#sampling). -It introduces additional _composite samplers_. Composite samplers use other samplers (_delegates_ or _children_) to make sampling decisions. The composite samplers invoke the delegate samplers, but eventually make the final call. +It introduces additional _composite samplers_. +Composite samplers use other samplers (_delegates_ or _children_) to make sampling decisions. +The composite samplers invoke the delegate samplers, but eventually make the final call. -The new samplers proposed here are mostly compatible with Consistent Probability Samplers. For detailed description of this concept see [probability sampling](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/tracestate-probability-sampling.md). However, the technical details in that document are outdated. For the current proposal see [OTEP 235](https://github.com/open-telemetry/oteps/blob/main/text/trace/0235-sampling-threshold-in-trace-state.md). Also see Draft PR 3910 [Probability Samplers based on W3C Trace Context Level 2](https://github.com/open-telemetry/opentelemetry-specification/pull/3910). +The new samplers proposed here are mostly compatible with Consistent Probability Samplers. For detailed description of this concept see [probability sampling (OTEP 235)](https://github.com/open-telemetry/oteps/blob/main/text/trace/0235-sampling-threshold-in-trace-state.md). +Also see Draft PR 3910 [Probability Samplers based on W3C Trace Context Level 2](https://github.com/open-telemetry/opentelemetry-specification/pull/3910). ## Motivation -The need for configuring head sampling has been explicitly or implicitly indicated in several discussions, both within the [Sampling SIG](https://docs.google.com/document/d/1gASMhmxNt9qCa8czEMheGlUW2xpORiYoD7dBD7aNtbQ) and in the wider community. Some of the discussions are going back a number of years, see for example +The need for configuring head sampling has been explicitly or implicitly indicated in several discussions, both within the [Sampling SIG](https://docs.google.com/document/d/1gASMhmxNt9qCa8czEMheGlUW2xpORiYoD7dBD7aNtbQ) and in the wider community. +Some of the discussions are going back a number of years, see for example - issue [173](https://github.com/open-telemetry/opentelemetry-specification/issues/173): Way to ignore healthcheck traces when using automatic tracer across all languages? - issue [1060](https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/1060): Exclude URLs from Tracing @@ -36,7 +40,8 @@ We present two quite different approaches to composite samplers. The first one u The second approach is applicable exclusively to Consistent Probability Samplers, but is more efficient and less prone to misconfiguration. It requires additional API to be provided by the delegate samplers. -__Note__: both approaches call for calculating unions of Attribute sets. Whenever such union is constructed, in case of conflicting attribute keys, the attribute definition from the last set that uses that key takes effect. Similarly, whenever modifications of `Tracestate` are chained, in case of conflicting keys, the last modification erases the previous values. +__Note__: both approaches call for calculating _unions_ of Attribute sets. +Whenever such union is constructed, in case of conflicting attribute keys, the attribute definition from the last set that uses that key takes effect. Similarly, whenever modifications of `Tracestate` are performed in sequence, in case of conflicting keys, the last modification erases the previous values. ## Approach One @@ -53,7 +58,7 @@ Upon invocation of its `shouldSample` method, it MUST go through the whole list - If all of the delegate Decisions are `DROP`, the composite sampler MUST return `DROP` Decision as well. If any of the delegate Decisions is `RECORD_AND_SAMPLE`, the composite sampler MUST return `RECORD_AND_SAMPLE` Decision. Otherwise, if any of the delegate Decisions is `RECORD_ONLY`, the composite sampler MUST return `RECORD_ONLY` Decision. -- The set of span Attributes to be added to the `Span` is the union of the sets of Attributes as provided by those delegate samplers which produced a sampling Decision other than `DROP`. +- The set of span `Attributes` to be added to the `Span` is the union of the sets of `Attributes` as provided by those delegate samplers which produced a sampling Decision other than `DROP`. - The `Tracestate` to be used with the new `Span` is obtained by cumulatively applying all the potential modifications of the parent `Tracestate` by the delegate samplers. However, the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry gets special handling as described below. If the final sampling Decision is `DROP` or `RECORD_ONLY`, the `th` entry MUST be removed. @@ -73,7 +78,7 @@ Upon invocation of its `shouldSample` method, it MUST go through the whole list - If all of the delegate Decisions are `RECORD_AND_SAMPLE`, the composite sampler MUST return `RECORD_AND_SAMPLE` Decision as well. If any of the delegate Decisions is `DROP`, the composite sampler MUST return `DROP` Decision. Otherwise, if any of the delegate Decisions is `RECORD_ONLY`, the composite sampler MUST return `RECORD_ONLY` Decision. -- If the resulting sampling Decision is `DROP`, the set of span Attributes to be added to the `Span` is empty. Otherwise, it is the union of the sets of Attributes as provided by the delegate samplers within their `SamplingResult`s. +- If the resulting sampling Decision is `DROP`, the set of span `Attributes` to be added to the `Span` is empty. Otherwise, it is the union of the sets of `Attributes` as provided by the delegate samplers within their `SamplingResult`s. - The `Tracestate` to be used with the new `Span` is obtained by cumulatively applying all the potential modfications of the parent `Tracestate` by the delegate samplers. However, the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry gets special handling as described below. If the final sampling Decision is `DROP` or `RECORD_ONLY`, the `th` entry MUST be removed. @@ -86,19 +91,28 @@ Each delegate sampler MUST be given a chance to participate in the sampling deci `Conjunction` is a composite sampler which takes two Samplers (delegates) as the arguments. These delegate samplers will be hereby referenced as First and Second. This kind of composition forms conditional chaining of both samplers. -Upon invocation of its `shouldSample` method, the Conjunction sampler MUST invoke `shouldSample` method on the First sampler, passing the same arguments as received, and examine the received sampling Decision. Upon receiving `DROP` or `RECORD_ONLY` decision it MUST return the SamplingResult from the First sampler, and it MUST NOT proceed with querying the Second sampler. If the sampling decision from the First sampler is `RECORD_AND_SAMPLE`, the Conjunction sampler MUST invoke `shouldSample` method on the Second sampler, effectively passing the `Tracestate` received from the First sampler as the parent trace state. +Upon invocation of its `shouldSample` method, the Conjunction sampler MUST invoke `shouldSample` method on the First sampler, passing the same arguments as received, and examine the received sampling Decision. +Upon receiving `DROP` or `RECORD_ONLY` decision it MUST return the `SamplingResult` (which includes a set of `Attributes` and `Tracestate` in addition to the sampling Decision) from the First sampler, and it MUST NOT proceed with querying the Second sampler. +If the sampling decision from the First sampler is `RECORD_AND_SAMPLE`, the Conjunction sampler MUST invoke `shouldSample` method on the Second sampler, effectively passing the `Tracestate` received from the First sampler as the parent trace state. If the sampling Decision from the Second sampler is `RECORD_AND_SAMPLE`, the Conjunction sampler MUST return a `SamplingResult` which is constructed as follows: - The sampling Decision is `RECORD_AND_SAMPLE`. -- The set of span Attributes to be added to the `Span` is the union of the sets of Attributes as provided by both samplers. +- The set of span `Attributes` to be added to the `Span` is the union of the sets of `Attributes` as provided by both samplers. - The `Tracestate` to be used with the new `Span` is as provided by the Second sampler, with special handling of the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry. -If the First sampler did not provide `th` entry in the returned `Tracestate`, or if the value of the corresponding `THRESHOLD` is not `0`, then the `th` entry MUST be removed from the resulting `Tracestate`. +If both First and Second samplers provided `th` entry in the returned `Tracestate`, the resulting `Tracestate` MUST contain `th` entry with the `THRESHOLD` being maximum of the `THRESHOLD`s provided by the First and Second samplers. +Otherwise, the `th` entry MUST be removed. -If the sampling Decision from the Second sampler is `DROP` or `RECORD_ONLY`, the Conjunction sampler MUST return a `SamplingResult` which is constructed as follows: +If the sampling Decision from the Second sampler is `RECORD_ONLY`, the Conjunction sampler MUST return a `SamplingResult` which is constructed as follows: + +- The sampling Decision is `RECORD_ONLY`. +- The set of span `Attributes` to be added to the `Span` is the set of `Attributes` returned by the First sampler. +- The `Tracestate` to be used with the new `Span` is the `Tracestate` provided by the Second sampler, but with the `th` entry removed. + +If the sampling Decision from the Second sampler is `DROP`, the Conjunction sampler MUST return a `SamplingResult` which is constructed as follows: - The sampling Decision is `DROP`. -- The set of span Attributes to be added to the `Span` is empty. +- The set of span `Attributes` to be added to the `Span` is empty. - The `Tracestate` to be used with the new `Span` is the `Tracestate` provided by the Second sampler, but with the `th` entry removed. The `Conjunction` sampler can be useful in a special case where the user wants to keep a group of traces, for example belonging to an end user session, together - meaning to make the same sampling decisions for all traces belonging to the group, as much as possible. @@ -106,13 +120,13 @@ One way of achieving this behavior for consistent probability samplers is to giv ### RuleBased -`RuleBased` is a composite sampler which performs `Span` categorization (e.g. when sampling decision depends on Span attributes) and sampling. +`RuleBased` is a composite sampler which performs `Span` categorization (e.g. when sampling decision depends on `Span` attributes) and sampling. The Spans can be grouped into separate categories, and each category can use a different Sampler. Categorization of Spans is aided by `Predicates`. #### Predicate -The Predicates represent logical expressions which can access Span Attributes (or anything else available when the sampling decision is to be made), and perform tests on the accessible values. +The Predicates represent logical expressions which can access `Span` `Attributes` (or anything else available when the sampling decision is to be made), and perform tests on the accessible values. For example, one can test if the target URL for a SERVER span matches a given pattern. `Predicate` interface allows users to create custom categories based on information that is available at the time of making the sampling decision. @@ -124,11 +138,11 @@ This is a routine/function/method for `Predicate`, which returns `true` if a giv The arguments represent the values that are made available for `ShouldSample`. -- `Context` with parent Span. -- `TraceId` of the Span to be created. -- Name of the Span to be created. -- Initial set of Attributes of the Span to be created. -- Collection of links that will be associated with the Span to be created. +- `Context` with parent `Span`. +- `TraceId` of the `Span` to be created. +- Name of the `Span` to be created. +- Initial set of `Attributes` of the `Span` to be created. +- Collection of links that will be associated with the `Span` to be created. #### Required Arguments for RuleBased @@ -154,7 +168,7 @@ S1 = RuleBased(ROOT, { }) ``` -Note: technically, `ROOT` is not a Span Kind, but is a special token matching all Spans with invalid parent context (i.e. the ROOT spans, regardless of their kind). +Note: technically, `ROOT` is not a `SpanKind`, but is a special token matching all Spans with invalid parent context (i.e. the ROOT spans, regardless of their kind). In the next step, we can build the sampler to handle non-root spans as well: @@ -220,36 +234,36 @@ A principle of operation for Approach Two is that `ShouldSample` is invoked only To make this approach possible, all Consistent Probability Samplers which participate in the samplers composition need to implement the following API, in addition to the standard Sampler API. We will use the term _Composable Sampler_ to denote Consistent Probability Samplers which provide the new API and conform to the rules described here. The composite samplers in Approach Two are Composable Samplers as well. -#### GetSamplingAdvice +#### GetSamplingIntent This is a routine/function/method for all Composable Samplers. Its purpose is to query the sampler about the activities it would perform had it been asked to make a sampling decision for a given span, however, without constructing the actual sampling Decision. -#### Required Arguments for GetSamplingAdvice: +#### Required Arguments for GetSamplingIntent: The arguments are the same as for [`ShouldSample`](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#shouldsample) except for the `TraceId`. -- `Context` with parent Span. -- Name of the Span to be created. +- `Context` with parent `Span`. +- Name of the `Span` to be created. - `SpanKind` of the `Span` to be created. -- Initial set of Attributes of the Span to be created. -- Collection of links that will be associated with the Span to be created. +- Initial set of `Attributes` of the `Span` to be created. +- Collection of links that will be associated with the `Span` to be created. #### Return value: -The return value is a structure (`SamplingAdvice`) with the following elements: +The return value is a structure (`SamplingIntent`) with the following elements: - The THRESHOLD value represented as a 14-character hexadecimal string, with value of `null` representing non-probabilistic `DROP` decision, -- A function (`GetAttributes`) that provides a set of Attributes to be added to the Span in case of positive final sampling decision, -- A function (`UpdateTraceState`) that given an input `Tracestate` and sampling Decision provides a `Tracestate` to be associated with the Span. The samplers SHOULD NOT add or modify the `th` value for the `ot` key within these functions. +- A function (`GetAttributes`) that provides a set of `Attributes` to be added to the `Span` in case of positive final sampling decision, +- A function (`UpdateTraceState`) that given an input `Tracestate` and sampling Decision provides a `Tracestate` to be associated with the `Span`. The samplers SHOULD NOT add or modify the `th` value for the `ot` key within these functions. #### Constructing `SamplingResult` The process of constructing the final `SamplingResult` in response to a call to `ShouldSample` on the root sampler of the composite samplers tree consists of the following steps. -- The sampler gets its own `SamplingAdvice`, it is a recursive process as described below (unless the sampler is a leaf), +- The sampler gets its own `SamplingIntent`, it is a recursive process as described below (unless the sampler is a leaf), - The sampler compares the received THRESHOLD value with the trace Randomness value to arrive at the final sampling `Decision`, -- In case of a positive sampling decision the sampler calls the received `GetAttributes` function to determine the set of Attributes to be added to the Span, in most cases it will be a recursive step, -- The sampler calls the received `UpdateTraceState` function passing the parent `Tracestate` and the final sampling `Decision` to get the new `Tracestate` to be associated with the Span - again, in most cases this is a recursive step, +- In case of a positive sampling decision the sampler calls the received `GetAttributes` function to determine the set of `Attributes` to be added to the `Span`, in most cases it will be a recursive step, +- The sampler calls the received `UpdateTraceState` function passing the parent `Tracestate` and the final sampling `Decision` to get the new `Tracestate` to be associated with the `Span` - again, in most cases this is a recursive step, - The sampler modifies (or removes) the `th` value for the `ot` key in the `Tracestate` according to the final sampling `Decision` and the THRESHOLD used in the second step above. ### ConsistentRuleBased @@ -261,21 +275,21 @@ This composite sampler re-uses the concept of Predicates from Approach One. - `SpanKind` - list of pairs (`Predicate`, `ComposableSampler`) -For calculating the `SamplingAdvice`, if the `Span` kind matches the specified kind, the sampler goes through the list in the provided order and calls `SpanMatches` on `Predicate`s passing the same arguments as received. If a call returns `true`, the result is as returned by `GetSamplingAdvice` called on the corresponding `ComposableSampler`. If the `SpanKind` does not match, or none of the calls to `SpanMatches` yield `true`, the result is obtained by calling `GetSamplingAdvice` on `ConsistentAlwaysOffSampler`. +For calculating the `SamplingIntent`, if the `Span` kind matches the specified kind, the sampler goes through the list in the provided order and calls `SpanMatches` on `Predicate`s passing the same arguments as received. If a call returns `true`, the result is as returned by `GetSamplingIntent` called on the corresponding `ComposableSampler`. If the `SpanKind` does not match, or none of the calls to `SpanMatches` yield `true`, the result is obtained by calling `GetSamplingIntent` on `ConsistentAlwaysOffSampler`. ### ConsistentAnyOf `ConsistentAnyOf` is a composite sampler which takes a non-empty list of ComposableSamplers (delegates) as the argument. The intention is to make a positive sampling decision if __any of__ the delegates would make a positive decision. -Upon invocation of its `GetSamplingAdvice` function, it MUST go through the whole list and invoke `GetSamplingAdvice` function on each delegate sampler, passing the same arguments as received. +Upon invocation of its `GetSamplingIntent` function, it MUST go through the whole list and invoke `GetSamplingIntent` function on each delegate sampler, passing the same arguments as received. -`ConsistentAnyOf` sampler MUST return a `SamplingAdvice` which is constructed as follows: +`ConsistentAnyOf` sampler MUST return a `SamplingIntent` which is constructed as follows: - If any of the delegates returned a non-`null` threshold value, the resulting threshold is the lexicographical minimum value from the set of those non-`null` values, otherwise `null`. -- The `GetAttributes` function calculates the union of Attribute sets as returned by the calls to `GetAttributes` function for each delegate (in the declared order) that returned a non-`null` threshold value, with later Attributes clobbering older Attributes with the same key. +- The `GetAttributes` function calculates the union of `Attribute` sets as returned by the calls to `GetAttributes` function for each delegate (in the declared order) that returned a non-`null` threshold value. - The `UpdateTraceState` function makes a chain of calls to the `UpdateTraceState` functions as returned by the delegates, passing the received `Tracestate` as argument to subsequent calls and returning the last value received. -Each delegate sampler MUST be given a chance to participate in calculating the `SamplingAdvice` as described above and MUST see the same argument values. The order of the delegate samplers does not affect the final sampling `Decision`. +Each delegate sampler MUST be given a chance to participate in calculating the `SamplingIntent` as described above and MUST see the same argument values. The order of the delegate samplers does not affect the final sampling `Decision`. ### ConsistentRateLimiting @@ -287,13 +301,13 @@ Each delegate sampler MUST be given a chance to participate in calculating the ` - maximum sampling (throughput) target rate The sampler SHOULD measure and keep the average rate of incoming spans, and therefore also of the desired ratio between the incoming span rate to the target span rate. -Upon invocation of its `GetSamplingAdvice` function, the composite sampler MUST get the `SamplingAdvice` from the delegate sampler, passing the same arguments as received. +Upon invocation of its `GetSamplingIntent` function, the composite sampler MUST get the `SamplingIntent` from the delegate sampler, passing the same arguments as received. -The returned `SamplingAdvice` is constructed as follows. +The returned `SamplingIntent` is constructed as follows. - If using the obtained threshold value as the final threshold would entail sampling more spans than the declared target rate, the sampler SHOULD increase the threshold to a value that would meet the target rate. Several algorithms can be used for threshold adjustment, no particular behavior is prescried by the specification though. -- The `GetAttributes` function returns the union of the set of Attributes returned by calling the delegate's `GetAttributes` and own Attributes. -- The `UpdateTraceState` function returns the `Tracestate` as returned by calling `UpdateTraceState` from the delegate's `SamplingAdvice`. +- The `GetAttributes` function returns the union of the set of `Attributes` returned by calling the delegate's `GetAttributes` and own `Attributes`. +- The `UpdateTraceState` function returns the `Tracestate` as returned by calling `UpdateTraceState` from the delegate's `SamplingIntent`. When using `ConsistentRateLimiting` in our requirements example as a replacement for `EachOf` and `RateLimiting`, we are left with no use case for any direct equivalent to `EachOf` in Approach Two. However, if such a need arises, `ConsistentEachOf` can be easily defined. @@ -302,7 +316,7 @@ However, if such a need arises, `ConsistentEachOf` can be easily defined. ### Example - sampling configuration with Approach Two -With the samplers introduced by Approach Two, our example requirements can be coded in a very similar way as with Approach One. However, the work of the samplers configured this way forms a tree of `GetSamplingAdvice` invocations rather than `ShouldSample` invocations as in Approach One. +With the samplers introduced by Approach Two, our example requirements can be coded in a very similar way as with Approach One. However, the work of the samplers configured this way forms a tree of `GetSamplingIntent` invocations rather than `ShouldSample` invocations as in Approach One. ``` S = ConsistentRateLimiting( @@ -331,7 +345,9 @@ Developers of Composable Samplers should consider that the sampling Decision the A number of composite samplers are already available as independent contributions ([RuleBasedRoutingSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/RuleBasedRoutingSampler.java), -[LinksBasedSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/LinksBasedSampler.java)). +[Stratified Sampling](https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/trace/stratified-sampling-example), +LinksBasedSampler [for Java](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/samplers/src/main/java/io/opentelemetry/contrib/sampler/LinksBasedSampler.java) +and [for DOTNET](https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/trace/links-based-sampler)). Also, historically, some Span categorization was introduced by [JaegerRemoteSampler](https://www.jaegertracing.io/docs/1.54/sampling/#remote-sampling). This proposal aims at generalizing these ideas, and at providing a bit more formal specification for the behavior of the composite samplers. From bfc3081e74e41c4814be330886b9ed9c55b52628 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Wed, 19 Jun 2024 09:09:59 -0700 Subject: [PATCH 25/30] Removed EachOf and the incorrect example using ConsistentRateLimiting sampler. --- text/0250-Composite_Samplers.md | 52 --------------------------------- 1 file changed, 52 deletions(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index 95f579931..2fd839d69 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -67,26 +67,6 @@ Otherwise, the resulting `Tracestate` MUST contain `th` entry with the `THRESHOL Each delegate sampler MUST be given a chance to participate in the sampling decision as described above and MUST see the same _parent_ state. The resulting sampling Decision does not depend on the order of the delegate samplers. -### EachOf - -`EachOf` is a composite sampler which takes a non-empty list of Samplers (delegates) as the argument. The intention is to make `RECORD_AND_SAMPLE` decision if __each of__ the delegates decides to `RECORD_AND_SAMPLE`. - -Upon invocation of its `shouldSample` method, it MUST go through the whole list and invoke `shouldSample` method on each delegate sampler, passing the same arguments as received, and collecting the delegates' sampling Decisions. - -`EachOf` sampler MUST return a `SamplingResult` with the following elements. - -- If all of the delegate Decisions are `RECORD_AND_SAMPLE`, the composite sampler MUST return `RECORD_AND_SAMPLE` Decision as well. -If any of the delegate Decisions is `DROP`, the composite sampler MUST return `DROP` Decision. -Otherwise, if any of the delegate Decisions is `RECORD_ONLY`, the composite sampler MUST return `RECORD_ONLY` Decision. -- If the resulting sampling Decision is `DROP`, the set of span `Attributes` to be added to the `Span` is empty. Otherwise, it is the union of the sets of `Attributes` as provided by the delegate samplers within their `SamplingResult`s. -- The `Tracestate` to be used with the new `Span` is obtained by cumulatively applying all the potential modfications of the parent `Tracestate` by the delegate samplers. However, the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry gets special handling as described below. - -If the final sampling Decision is `DROP` or `RECORD_ONLY`, the `th` entry MUST be removed. -If the sampling Decision is `RECORD_AND_SAMPLE`, and there's no `th` entry in any of the `Tracestate` provided by the delegates, the `th` entry MUST be also removed. -Otherwise, the resulting `Tracestate` MUST contain `th` entry with the `THRESHOLD` value being the maximum of all the `THRESHOLD` values as reported the delegates. - -Each delegate sampler MUST be given a chance to participate in the sampling decision as described above and MUST see the same _parent_ state. The resulting sampling Decision does not depend on the order of the delegate samplers. - ### Conjunction `Conjunction` is a composite sampler which takes two Samplers (delegates) as the arguments. These delegate samplers will be hereby referenced as First and Second. This kind of composition forms conditional chaining of both samplers. @@ -188,35 +168,6 @@ Finally, the last step is to put a limit on the stream of exported spans. One of S4 = Conjunction(S3, RateLimitingSampler(1000 * 60)) ``` -### Example - sampling configuration 2 - -Many users are interested in [Consistent Probability Sampling](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#consistent-probability-sampler) (CP), as it gives them a chance to calculate span-based metrics even when sampling is active. The configuration presented above uses the traditional samplers, which do not offer this benefit. - -Here is how an equivalent configuration can be put together using [CP samplers](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/tracestate-probability-sampling.md#samplers). In this example, the following implementations are used: - -- [ConsistentAlwaysOffSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/consistent-sampling/src/main/java/io/opentelemetry/contrib/sampler/consistent56/ConsistentAlwaysOffSampler.java) -- [ConsistentAlwaysOnSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/consistent-sampling/src/main/java/io/opentelemetry/contrib/sampler/consistent56/ConsistentAlwaysOnSampler.java) -- [ConsistentFixedThresholdSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/consistent-sampling/src/main/java/io/opentelemetry/contrib/sampler/consistent56/ConsistentFixedThresholdSampler.java) -- [ConsistentParentBasedSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/consistent-sampling/src/main/java/io/opentelemetry/contrib/sampler/consistent56/ConsistentParentBasedSampler.java) -- [ConsistentRateLimitingSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/consistent-sampling/src/main/java/io/opentelemetry/contrib/sampler/consistent56/ConsistentRateLimitingSampler.java) - -``` -S = EachOf( - AnyOf( - ConsistentParentBased( - RuleBased(ROOT, { - (http.target == /healthcheck) => ConsistentAlwaysOff, - (http.target == /checkout) => ConsistentAlwaysOn, - true => ConsistentFixedThreshold(0.25) - }), - RuleBased(CLIENT, { - (http.url == /foo) => ConsistentAlwaysOn - } - ), - ConsistentRateLimiting(1000 * 60) - ) -``` - ### Limitations of composite samplers in Approach One Not all samplers can participate as components of composite samplers without undesired or unexpected effects. Some samplers require that they _see_ each `Span` being created, even if the span is going to be dropped. Some samplers update the trace state or maintain internal state, and for their correct behavior it it is assumed that their sampling decisions will be honored by the tracer at the face value in all cases. A good example for this are rate limiting samplers which have to keep track of the rate of created spans and/or the rate of positive sampling decisions. @@ -309,9 +260,6 @@ The returned `SamplingIntent` is constructed as follows. - The `GetAttributes` function returns the union of the set of `Attributes` returned by calling the delegate's `GetAttributes` and own `Attributes`. - The `UpdateTraceState` function returns the `Tracestate` as returned by calling `UpdateTraceState` from the delegate's `SamplingIntent`. -When using `ConsistentRateLimiting` in our requirements example as a replacement for `EachOf` and `RateLimiting`, we are left with no use case for any direct equivalent to `EachOf` in Approach Two. -However, if such a need arises, `ConsistentEachOf` can be easily defined. - ## Summary - Approach Two ### Example - sampling configuration with Approach Two From ed2a314c94ab12f8e2f711511579e0d373870900 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Thu, 20 Jun 2024 17:18:31 -0700 Subject: [PATCH 26/30] Removing special handling of THRESHOLD from Approach One. --- text/0250-Composite_Samplers.md | 34 +++++++++++++-------------------- 1 file changed, 13 insertions(+), 21 deletions(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index 2fd839d69..6c7947576 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -5,7 +5,7 @@ It introduces additional _composite samplers_. Composite samplers use other samplers (_delegates_ or _children_) to make sampling decisions. The composite samplers invoke the delegate samplers, but eventually make the final call. -The new samplers proposed here are mostly compatible with Consistent Probability Samplers. For detailed description of this concept see [probability sampling (OTEP 235)](https://github.com/open-telemetry/oteps/blob/main/text/trace/0235-sampling-threshold-in-trace-state.md). +Some of the new samplers proposed here have been designed to work with Consistent Probability Samplers. For detailed description of this concept see [probability sampling (OTEP 235)](https://github.com/open-telemetry/oteps/blob/main/text/trace/0235-sampling-threshold-in-trace-state.md). Also see Draft PR 3910 [Probability Samplers based on W3C Trace Context Level 2](https://github.com/open-telemetry/opentelemetry-specification/pull/3910). ## Motivation @@ -36,9 +36,10 @@ Head-based sampling requirements. - however, capture all calls to service `/foo` (even if the trace will be incomplete) - in any case, do not exceed 1000 spans/minute -We present two quite different approaches to composite samplers. The first one uses only the current [sampling API](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#sampling). It can be applied to a large variety of samplers, but is not very efficient nor elegant. +We present two quite different approaches to composite samplers. The first one uses only the current [sampling API](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#sampling). +It can be applied to a large variety of samplers, but may not work correctly for Consistent Probability Samplers. It is also not very efficient nor elegant. -The second approach is applicable exclusively to Consistent Probability Samplers, but is more efficient and less prone to misconfiguration. It requires additional API to be provided by the delegate samplers. +The second approach is applicable exclusively to Consistent Probability Samplers, and is more efficient and less prone to misconfiguration. It requires additional API to be provided by the delegate samplers. __Note__: both approaches call for calculating _unions_ of Attribute sets. Whenever such union is constructed, in case of conflicting attribute keys, the attribute definition from the last set that uses that key takes effect. Similarly, whenever modifications of `Tracestate` are performed in sequence, in case of conflicting keys, the last modification erases the previous values. @@ -59,11 +60,7 @@ Upon invocation of its `shouldSample` method, it MUST go through the whole list If any of the delegate Decisions is `RECORD_AND_SAMPLE`, the composite sampler MUST return `RECORD_AND_SAMPLE` Decision. Otherwise, if any of the delegate Decisions is `RECORD_ONLY`, the composite sampler MUST return `RECORD_ONLY` Decision. - The set of span `Attributes` to be added to the `Span` is the union of the sets of `Attributes` as provided by those delegate samplers which produced a sampling Decision other than `DROP`. -- The `Tracestate` to be used with the new `Span` is obtained by cumulatively applying all the potential modifications of the parent `Tracestate` by the delegate samplers. However, the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry gets special handling as described below. - -If the final sampling Decision is `DROP` or `RECORD_ONLY`, the `th` entry MUST be removed. -If the sampling Decision is `RECORD_AND_SAMPLE`, and there's no `th` entry in any of the `Tracestate` provided by the delegates that decided to `RECORD_AND_SAMPLE`, the `th` entry MUST be also removed. -Otherwise, the resulting `Tracestate` MUST contain `th` entry with the `THRESHOLD` value being the minimum of all the `THRESHOLD` values as reported by those delegates that decided to `RECORD_AND_SAMPLE`. +- The `Tracestate` to be used with the new `Span` is obtained by cumulatively applying all the potential modifications of the parent `Tracestate` by the delegate samplers. Each delegate sampler MUST be given a chance to participate in the sampling decision as described above and MUST see the same _parent_ state. The resulting sampling Decision does not depend on the order of the delegate samplers. @@ -79,24 +76,19 @@ If the sampling Decision from the Second sampler is `RECORD_AND_SAMPLE`, the Con - The sampling Decision is `RECORD_AND_SAMPLE`. - The set of span `Attributes` to be added to the `Span` is the union of the sets of `Attributes` as provided by both samplers. -- The `Tracestate` to be used with the new `Span` is as provided by the Second sampler, with special handling of the `th` sub-key (the sampling rejection `THRESHOLD`) for the `ot` entry. -If both First and Second samplers provided `th` entry in the returned `Tracestate`, the resulting `Tracestate` MUST contain `th` entry with the `THRESHOLD` being maximum of the `THRESHOLD`s provided by the First and Second samplers. -Otherwise, the `th` entry MUST be removed. +- The `Tracestate` to be used with the new `Span` is as provided by the Second sampler. If the sampling Decision from the Second sampler is `RECORD_ONLY`, the Conjunction sampler MUST return a `SamplingResult` which is constructed as follows: - The sampling Decision is `RECORD_ONLY`. - The set of span `Attributes` to be added to the `Span` is the set of `Attributes` returned by the First sampler. -- The `Tracestate` to be used with the new `Span` is the `Tracestate` provided by the Second sampler, but with the `th` entry removed. +- The `Tracestate` to be used with the new `Span` is the `Tracestate` provided by the Second sampler. If the sampling Decision from the Second sampler is `DROP`, the Conjunction sampler MUST return a `SamplingResult` which is constructed as follows: - The sampling Decision is `DROP`. - The set of span `Attributes` to be added to the `Span` is empty. -- The `Tracestate` to be used with the new `Span` is the `Tracestate` provided by the Second sampler, but with the `th` entry removed. - -The `Conjunction` sampler can be useful in a special case where the user wants to keep a group of traces, for example belonging to an end user session, together - meaning to make the same sampling decisions for all traces belonging to the group, as much as possible. -One way of achieving this behavior for consistent probability samplers is to give all traces belonging to the group the same _randomness_ (represented by `r-value`), based on some criteria shared by all traces belonging to the group. This can be done by a special sampler which would provide the required `r-value` for all `ROOT` spans of the involved traces. When using such a sampler as the First delegate for `Conjunction`, this functionality can be encapsulated in a separate sampler, without making any changes to the current SDK specification. +- The `Tracestate` to be used with the new `Span` is the `Tracestate` provided by the Second sampler. ### RuleBased @@ -135,7 +127,7 @@ The order of `Predicate`s is essential. If more than one `Predicate` matches a ` ## Summary - Approach One -### Example - sampling configuration 1 +### Example - sampling configuration Going back to our example of sampling requirements, we can now configure the head sampler to support this particular case, using an informal notation of samplers and their arguments. First, let's express the requirements for the ROOT spans as follows. @@ -172,8 +164,6 @@ S4 = Conjunction(S3, RateLimitingSampler(1000 * 60)) Not all samplers can participate as components of composite samplers without undesired or unexpected effects. Some samplers require that they _see_ each `Span` being created, even if the span is going to be dropped. Some samplers update the trace state or maintain internal state, and for their correct behavior it it is assumed that their sampling decisions will be honored by the tracer at the face value in all cases. A good example for this are rate limiting samplers which have to keep track of the rate of created spans and/or the rate of positive sampling decisions. -A special attention is required for CP samplers. The sampling probability they record in trace-state is later used to calculate [_adjusted count_](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#adjusted-count), which, in turn, is used to calculate span-based metrics. While the composite samplers presented here are compatible with CP samplers, generally, mixing CP samplers with other types of samplers may lead to undefined or sometimes incorrect adjusted counts. - The need to encode and decode the `Tracestate` multiple times affects performance of the composite samplers. This drawback is eliminated in Approach Two. ## Approach Two @@ -256,10 +246,12 @@ Upon invocation of its `GetSamplingIntent` function, the composite sampler MUST The returned `SamplingIntent` is constructed as follows. -- If using the obtained threshold value as the final threshold would entail sampling more spans than the declared target rate, the sampler SHOULD increase the threshold to a value that would meet the target rate. Several algorithms can be used for threshold adjustment, no particular behavior is prescried by the specification though. +- If using the obtained threshold value as the final threshold would entail sampling more spans than the declared target rate, the sampler SHOULD increase the threshold to a value that would meet the target rate. Several algorithms can be used for threshold adjustment, no particular behavior is prescribed by the specification though. - The `GetAttributes` function returns the union of the set of `Attributes` returned by calling the delegate's `GetAttributes` and own `Attributes`. - The `UpdateTraceState` function returns the `Tracestate` as returned by calling `UpdateTraceState` from the delegate's `SamplingIntent`. +TO DO: consider introducing a `ConsistentConjuntion` sampler (similar to `Conjunction` from Approach One) that would generalize the relationship between the delegate and the principal sampler, and remove the explicit delegate from `ConsistentRateLimiting`. + ## Summary - Approach Two ### Example - sampling configuration with Approach Two @@ -287,7 +279,7 @@ S = ConsistentRateLimiting( Making sampling decisions with samplers from Approach Two is more efficient than in Approach One, especially if, platform permitting, `null` values can be used for `GetAttributes` and `UpdateTraceState` functions to represent the prevailing trivial cases of _no-new-attributes_ and _no-special-trace-state-keys_. The only limitation of this approach that it operates exclusively within the domain of Composable Samplers (a subset of Consistent Probability Samplers). -Developers of Composable Samplers should consider that the sampling Decision they advise might be different from the final sampling Decision. +Developers of Composable Samplers should consider that the sampling Decision they declare as their intent might be different from the final sampling Decision. ## Prior art From 9c47d0f99407c6093443902ff32853e907aa29ce Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Fri, 26 Jul 2024 16:50:26 -0700 Subject: [PATCH 27/30] Minor changes, whitespace fixes. --- text/0250-Composite_Samplers.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index 6c7947576..910195bbd 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -66,7 +66,7 @@ Each delegate sampler MUST be given a chance to participate in the sampling deci ### Conjunction -`Conjunction` is a composite sampler which takes two Samplers (delegates) as the arguments. These delegate samplers will be hereby referenced as First and Second. This kind of composition forms conditional chaining of both samplers. +`Conjunction` is a composite sampler which takes two Samplers (delegates) as the arguments. These delegate samplers will be hereby referenced as First and Second. This kind of composition forms conditional chaining of both samplers. Upon invocation of its `shouldSample` method, the Conjunction sampler MUST invoke `shouldSample` method on the First sampler, passing the same arguments as received, and examine the received sampling Decision. Upon receiving `DROP` or `RECORD_ONLY` decision it MUST return the `SamplingResult` (which includes a set of `Attributes` and `Tracestate` in addition to the sampling Decision) from the First sampler, and it MUST NOT proceed with querying the Second sampler. @@ -179,7 +179,7 @@ The composite samplers in Approach Two are Composable Samplers as well. This is a routine/function/method for all Composable Samplers. Its purpose is to query the sampler about the activities it would perform had it been asked to make a sampling decision for a given span, however, without constructing the actual sampling Decision. -#### Required Arguments for GetSamplingIntent: +#### Required Arguments for GetSamplingIntent The arguments are the same as for [`ShouldSample`](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#shouldsample) except for the `TraceId`. @@ -189,11 +189,11 @@ The arguments are the same as for [`ShouldSample`](https://github.com/open-telem - Initial set of `Attributes` of the `Span` to be created. - Collection of links that will be associated with the `Span` to be created. -#### Return value: +#### Return value The return value is a structure (`SamplingIntent`) with the following elements: -- The THRESHOLD value represented as a 14-character hexadecimal string, with value of `null` representing non-probabilistic `DROP` decision, +- The THRESHOLD value represented as a 14-character hexadecimal string, with value of `null` representing non-probabilistic `DROP` decision (implementations MAY use different representation, if it appears more performant or convenient), - A function (`GetAttributes`) that provides a set of `Attributes` to be added to the `Span` in case of positive final sampling decision, - A function (`UpdateTraceState`) that given an input `Tracestate` and sampling Decision provides a `Tracestate` to be associated with the `Span`. The samplers SHOULD NOT add or modify the `th` value for the `ot` key within these functions. @@ -205,7 +205,7 @@ The process of constructing the final `SamplingResult` in response to a call to - The sampler compares the received THRESHOLD value with the trace Randomness value to arrive at the final sampling `Decision`, - In case of a positive sampling decision the sampler calls the received `GetAttributes` function to determine the set of `Attributes` to be added to the `Span`, in most cases it will be a recursive step, - The sampler calls the received `UpdateTraceState` function passing the parent `Tracestate` and the final sampling `Decision` to get the new `Tracestate` to be associated with the `Span` - again, in most cases this is a recursive step, -- The sampler modifies (or removes) the `th` value for the `ot` key in the `Tracestate` according to the final sampling `Decision` and the THRESHOLD used in the second step above. +- The sampler modifies (or removes) the `th` value for the `ot` key in the `Tracestate` according to the final sampling `Decision` and the THRESHOLD used in the second step above. ### ConsistentRuleBased From 05943e91fd578f255056ade9d53afa62f855ed53 Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Wed, 28 Aug 2024 13:58:06 -0700 Subject: [PATCH 28/30] Fine tuning the behavior of GetAttributes for AnyOf sampler. --- text/0250-Composite_Samplers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index 910195bbd..5a1eb4dfc 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -227,7 +227,7 @@ Upon invocation of its `GetSamplingIntent` function, it MUST go through the whol `ConsistentAnyOf` sampler MUST return a `SamplingIntent` which is constructed as follows: - If any of the delegates returned a non-`null` threshold value, the resulting threshold is the lexicographical minimum value from the set of those non-`null` values, otherwise `null`. -- The `GetAttributes` function calculates the union of `Attribute` sets as returned by the calls to `GetAttributes` function for each delegate (in the declared order) that returned a non-`null` threshold value. +- The `GetAttributes` function calculates the union of `Attribute` sets as returned by the calls to `GetAttributes` function for each delegate, in the declared order. - The `UpdateTraceState` function makes a chain of calls to the `UpdateTraceState` functions as returned by the delegates, passing the received `Tracestate` as argument to subsequent calls and returning the last value received. Each delegate sampler MUST be given a chance to participate in calculating the `SamplingIntent` as described above and MUST see the same argument values. The order of the delegate samplers does not affect the final sampling `Decision`. From 4c40e85e21f78bb0ebb7c49f8c1bd71b95199f7e Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Thu, 29 Aug 2024 17:38:20 -0700 Subject: [PATCH 29/30] Added link for the PR with the prototype implementation for Java. --- text/0250-Composite_Samplers.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index 5a1eb4dfc..697e6d97c 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -281,6 +281,10 @@ Making sampling decisions with samplers from Approach Two is more efficient than Developers of Composable Samplers should consider that the sampling Decision they declare as their intent might be different from the final sampling Decision. +### Prototyping + +A prototype implementation of ComposableSamplers for Java is available for review at [PR 1443](https://github.com/open-telemetry/opentelemetry-java-contrib/pull/1443). + ## Prior art A number of composite samplers are already available as independent contributions From 577537e09b4e2ae57d75b7a699a7ddced2a75b7f Mon Sep 17 00:00:00 2001 From: Peter Findeisen Date: Thu, 19 Sep 2024 15:41:55 -0700 Subject: [PATCH 30/30] Updated the link to prototype implementation. --- text/0250-Composite_Samplers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0250-Composite_Samplers.md b/text/0250-Composite_Samplers.md index 697e6d97c..49debadf3 100644 --- a/text/0250-Composite_Samplers.md +++ b/text/0250-Composite_Samplers.md @@ -283,7 +283,7 @@ Developers of Composable Samplers should consider that the sampling Decision the ### Prototyping -A prototype implementation of ComposableSamplers for Java is available for review at [PR 1443](https://github.com/open-telemetry/opentelemetry-java-contrib/pull/1443). +A prototype implementation of ComposableSamplers for Java is available, see [ConsistentSampler](https://github.com/open-telemetry/opentelemetry-java-contrib/blob/main/consistent-sampling/src/main/java/io/opentelemetry/contrib/sampler/consistent56/ConsistentSampler.java) and its subclasses. ## Prior art