diff --git a/docs/specs/om/open_metrics_spec_2_0.md b/docs/specs/om/open_metrics_spec_2_0.md index 675b3d0b3..8a6c36b65 100644 --- a/docs/specs/om/open_metrics_spec_2_0.md +++ b/docs/specs/om/open_metrics_spec_2_0.md @@ -62,7 +62,19 @@ This section MUST be read together with the ABNF section. In case of disagreemen #### Values -Metric values in OpenMetrics MUST be either floating points or integers. Note that ingestors of the format MAY only support float64. The non-real values NaN, +Inf and -Inf MUST be supported. NaN MUST NOT be considered a missing value, but it MAY be used to signal a division by zero. +Metric values in OpenMetrics MUST be either numbers or complex data types. + +Numbers MUST be either floating points or integers. Note that ingestors of the format MAY only support float64. The non-real values NaN, +Inf and -Inf MUST be supported. NaN MUST NOT be considered a missing value, but it MAY be used to signal a division by zero. + +Complex data types MUST contain all information necessary to recreate a sample of a Metric Type, with the exception of Created Value and Exemplars. + +List of complex data types: +- Integer counter native histograms for the Metric Type Histogram. +- Float counter native histograms for the Metric Type Histogram. +- Integer gauge native histograms for the Metric Type GaugeHistogram. +- Float gauge native histograms for the Metric Type GaugeHistogram. + +Complex data types MUST occur only in the corresponding MetricFamily. This means for example that a counter cannot have an integer counter native histogram value. ##### Booleans @@ -216,42 +228,103 @@ MetricFamilies of type Info MUST have an empty Unit string. Histograms measure distributions of discrete events. Common examples are the latency of HTTP requests, function runtimes, or I/O request sizes. -A Histogram MetricPoint MUST contain at least one bucket, and SHOULD contain Sum, and Created values. Every bucket MUST have a threshold and a value. +A Histogram MetricPoint MUST contain Count, Sum values. + +The Count value MUST be equal to the number of measurements taken by the Histogram. The Count is a counter semantically. The Count MUST be an integer and MUST NOT be NaN or negative. -Histogram MetricPoints MUST have one bucket with an +Inf threshold. Buckets MUST be cumulative. As an example for a metric representing request latency in seconds its values for buckets with thresholds 1, 2, 3, and +Inf MUST follow value_1 <= value_2 <= value_3 <= value_+Inf. If ten requests took 1 second each, the values of the 1, 2, 3, and +Inf buckets MUST equal 10. +The Sum value MUST be equal to the Sum of all the measured event values. The Sum is only a counter semantically as long as there are no negative event values measured by the Histogram MetricPoint. -The +Inf bucket counts all requests. If present, the Sum value MUST equal the Sum of all the measured event values. Bucket thresholds within a MetricPoint MUST be unique. +A Histogram MUST measure non NaN values in either [classic buckets](#classic-buckets) or [native buckets](#native-buckets) or both. If a Histogram stops measuring values in either classic or native buckets and keeps measuring values in the other, it MUST clear and not expose the buckets it stopped measuring into. This avoids exposing different distribution from the two kind of buckets at the same time. -Semantically, Sum, and buckets values are counters so MUST NOT be NaN or negative. -Negative threshold buckets MAY be used, but then the Histogram MetricPoint MUST NOT contain a sum value as it would no longer be a counter semantically. Bucket thresholds MUST NOT equal NaN. Count and bucket values MUST be integers. +Every bucket MUST have well defined boundaries and a value. Boundaries of a bucket MUST NOT be NaN. Bucket values MUST be integers. Semantically, bucket values are counters so MUST NOT be NaN or negative. + +A Histogram SHOULD refuse to measure NaN value as adding NaN to the Sum will make the Sum equal to NaN and mask the sum of the real measurements until the next reset of the counters. If a Histogram does allow NaN, then NaN MUST be counted in the Count and MUST be added to the Sum, resulting in the Sum becoming NaN. + +A Histogram MAY refuse to measure +Inf and -Inf values as adding these to the Sum will mask the sum of the real measurements until the next reset of the counters. If a Histogram measures +Inf or -Inf, then +Inf or -Inf MUST be counted in the Count and MUST be added to the Sum, potentially resulting in +Inf, -Inf or NaN in the Sum, the later for example in case of adding +Inf to -Inf. A Histogram MetricPoint SHOULD have a Timestamp value called Created. This can help ingestors discern between new metrics and long-running ones it did not see before. -A Histogram's Metric's LabelSet MUST NOT have a "le" label name. +If the Histogram Metric has MetricPoints with classic buckets, the Histogram's Metric's LabelSet MUST NOT have a "le" label name. -Bucket values MAY have exemplars. Buckets are cumulative to allow monitoring systems to drop any non-+Inf bucket for performance/anti-denial-of-service reasons in a way that loses granularity but is still a valid Histogram. +##### Classic buckets - +Every classic bucket MUST have a threshold. Classic bucket thresholds within a MetricPoint MUST be unique. + +A classic bucket MUST cover every measured value less or equal to its threshold, or to put it another way, classic buckets MUST be cumulative. Classic buckets are cumulative to allow monitoring systems to drop any non-+Inf bucket for performance/anti-denial-of-service reasons in a way that loses granularity but is still a valid Histogram. + +As an example for a metric representing request latency in seconds its values for classic buckets with thresholds 1, 2, 3, and +Inf MUST follow value_1 <= value_2 <= value_3 <= value_+Inf. If ten requests took 1 second each, the values of the 1, 2, 3, and +Inf buckets MUST equal 10. + +Histogram MetricPoints with classic buckets MUST have one classic bucket with a +Inf threshold. The +Inf bucket counts all requests. + +The Count value MUST be equal to the value of the +Inf bucket. + +If the NaN value is allowed, it MUST be counted in the +Inf bucket, and MUST not be counted in any other bucket. The ratonale is that NaN does not belong to any bucket mathematically, however instrumentation libraries traditionally put it into the +Inf bucket. + +Negative threshold classic buckets MAY be used. + +Classic bucket values MAY have exemplars. The value of the exemplar MUST be within the classic bucket. Exemplars SHOULD be put into the classic bucket with the lowest threshold that includes the exemplar value. A classic bucket MUST NOT have more than one exemplar. + +##### Native buckets + +Histogram MetricPoints with native buckets MUST have a Schema value. The Schema is an 8 bit signed integer between -4 and 8. Schemas between -9 and 52 are called Standard (exponential) Schemas, the currently unused Schemas -9 to -5 and 9 to 52 are reserved to be used as Standard Schemas later. + +For any Standard Schema n, the Histogram MetricPoint MAY contain positive, negative native buckets and MUST contain a zero native bucket. Empty positive or negative native buckets SHOULD NOT be present. + +As the Standard Schema allows for many positive and negative native buckets, the Histogram SHOULD have strategies to reset itself to empty when the number of native buckets is too high. Such strategies are out of scope for this specification. + +In case of Standard Schemas, the boundaries of a positive or negative native bucket with index i MUST be calculated as follows (using Python syntax): + +The upper inclusive limit of a positive native bucket: `(2**2**-n)**i` + +The lower exclusive limit of a positive native bucket: `(2**2**-n)**(i-1)` + +The lower inclusive limit of a negative native bucket: `-((2**2**-n)**i)` + +The upper exclusive limit of a negative native bucket: `-((2**2**-n)**(i-1))` + +i is an integer number that MAY be negative. + +There are exceptions to the rules above concerning the largest and smallest finite values representable as a float64 (called MaxFloat64 and MinFloat64 in the following) and the positive and negative infinity values (+Inf and -Inf): + +The positive native bucket that contains MaxFloat64 (according to the boundary formulas above) has an upper inclusive limit of MaxFloat64 (rather than the limit calculated by the formulas above, which would overflow float64). + +The next positive native bucket (index i+1 relative to the bucket from the previous item) has a lower exclusive limit of MaxFloat64 and an upper inclusive limit of +Inf. (It could be called a positive native overflow bucket.) + +The negative native bucket that contains MinFloat64 (according to the boundary formulas above) has a lower inclusive limit of MinFloat64 (rather than the limit calculated by the formulas above, which would underflow float64). + +The next negative native bucket (index i+1 relative to the bucket from the previous item) has an upper exclusive limit of MinFloat64 and an lower inclusive limit of -Inf. (It could be called a negative native overflow bucket.) + +Native buckets beyond the +Inf and -Inf buckets described above MUST NOT be used. + +If the zero native bucket is present, the Historam MetricPoint MUST have a Zero threshold. The Zero threshold MUST be a non-negative float64 value (threshold >= 0.0). The boundaries of the zero native bucket are `[-threshold, threshold]` inclusively. + +If the zero native bucket is present, any measured value that falls into the zero native bucket MUST be counted towards the zero native bucket and MUST NOT be counted in any other native bucket. The Zero threshold SHOULD be equal to a lower limit of an arbitrary native bucket. + +If the NaN value is not allowed, then the Count value MUST be equal to the sum of the negative, positive and zero native buckets. + +If the NaN value is allowed, it MUST NOT be counted in any bucket and the Count MUST be greater than the sum of the negative, positive and zero native buckets. The ratonale is that NaN does not belong to any bucket mathematically, and instrumentation libraries traditionally don't put it into any bucket. + +A Histogram MetricPoint with native buckets MAY contain exemplars. + +Exemplars associated with a Histogram MetricPoint with native buckets SHOULD have a timestamp. Note: storage implementations may drop exemplars without timestamps if keeping track of exemplars without timestamps is too resource hungry. -Each bucket covers the values less and or equal to it, and the value of the exemplar MUST be within this range. Exemplars SHOULD be put into the bucket with the highest value. A bucket MUST NOT have more than one exemplar. +The values of exemplars in a Histogram MetricPoint with native buckets SHOULD be evenly distributed to avoid only representing the bucket with the highest value and therefore most common case. #### GaugeHistogram GaugeHistograms measure current distributions. Common examples are how long items have been waiting in a queue, or size of the requests in a queue. -A GaugeHistogram MetricPoint MUST have one bucket with an +Inf threshold, and SHOULD contain a Gsum value. Every bucket MUST have a threshold and a value. +A GaugeHistogram MetricPoint MUST contain either classic buckets or native buckets or both. -The buckets for a GaugeHistogram follow all the same rules as for a Histogram. +A GaugeHistogram MetricPoint SHOULD contain Gcount, Gsum. Every bucket MUST have well defined boundaries and a value. Boundaries of a bucket MUST NOT be NaN. Gcount and bucket values MUST be integers. -The bucket and Gsum of a GaugeHistogram are conceptually gauges, however bucket values MUST NOT be negative or NaN. If negative threshold buckets are present, then sum MAY be negative. Gsum MUST NOT be NaN. Bucket values MUST be integers. +The bucket and Gsum of a GaugeHistogram are conceptually gauges, however bucket values MUST NOT be negative or NaN. If negative threshold buckets are present, then Gsum MAY be negative. Gsum MUST NOT be NaN. Bucket values MUST be integers. A GaugeHistogram's Metric's LabelSet MUST NOT have a "le" label name. -Bucket values can have exemplars. +The buckets for a GaugeHistogram follow all the same rules as for a Histogram, with Gcount playing the same role as Count. -Each bucket covers the values less and or equal to it, and the value of the exemplar MUST be within this range. Exemplars SHOULD be put into the bucket with the highest value. A bucket MUST NOT have more than one exemplar. +The exemplars for a GaugeHistogram follow all the same rules as for a Histogram. #### Summary @@ -322,6 +395,7 @@ metric-type = counter / gauge / histogram / gaugehistogram / stateset metric-type =/ info / summary / unknown sample = metricname [labels] SP number [SP timestamp] [exemplar] LF +sample =/ metricname [labels] SP "{" complextype "}" [SP timestamp] *exemplar LF exemplar = SP HASH SP labels SP number [SP timestamp] @@ -383,6 +457,58 @@ escaped-char =/ BS normal-char ; Any unicode character, except newline, double quote, and backslash normal-char = %x00-09 / %x0B-21 / %x23-5B / %x5D-D7FF / %xE000-10FFFF + +; Complex types +complextype = nativehistogram + +nativehistogram = nh-count "," nh-sum "," nh-schema "," nh-zero-threshold "," nh-zero-count [ "," nh-negative-spans "," nh-negative-buckets ] [ "," nh-positive-spans "," nh-positive-buckets ] +nativehistogram =/ nh-f-count "," nh-sum "," nh-schema "," nh-zero-threshold "," nh-f-zero-count [ "," nh-negative-spans "," nh-f-negative-buckets ] [ "," nh-positive-spans "," nh-f-positive-buckets ] + +; count:x +nh-count = %d99.111.117.110.116 ":" non-negative-integer +nh-f-count = %d99.111.117.110.116 ":" float-format-number +; sum:f allows real numbers and +-Inf and NaN +nh-sum = %d115.117.109 ":" number +; schema:i +nh-schema = %d115.99.104.101.109.97 ":" integer +; zero_threshold:f +nh-zero-threshold = %d122.101.114.111 "_" %d116.104.114.101.115.104.111.108.100 ":" realnumber +; zero_count:x +nh-zero-count = %d122.101.114.111 "_" %d99.111.117.110.116 ":" non-negative-integer +nh-f-zero-count = %d122.101.114.111 "_" %d99.111.117.110.116 ":" float-format-number +; negative_spans:[1:2,3:4] and negative_spans:[] +nh-negative-spans = %d110.101.103.97.116.105.118.101 "_" %d115.112.97.110.115 ":" "[" [nh-spans] "]" +nh-positive-spans = %d112.111.115.105.116.105.118.101 "_" %d115.112.97.110.115 ":" "[" [nh-spans] "]" +; Spans can start from any index, even negative, however subsequent spans +; can only advance the index, not decrease it. +nh-spans = nh-start-span *("," nh-span) +nh-start-span = integer ":" positive-integer +nh-span = non-negative-integer ":" positive-integer + +nh-negative-buckets = %d110.101.103.97.116.105.118.101 "_" %d98.117.99.107.101.116.115 ":" "[" [nh-buckets] "]" +nh-positive-buckets = %d112.111.115.105.116.105.118.101 "_" %d98.117.99.107.101.116.115 ":" "[" [nh-buckets] "]" +nh-f-negative-buckets = %d110.101.103.97.116.105.118.101 "_" %d98.117.99.107.101.116.115 ":" "[" [nh-f-buckets] "]" +nh-f-positive-buckets = %d112.111.115.105.116.105.118.101 "_" %d98.117.99.107.101.116.115 ":" "[" [nh-f-buckets] "]" + +nh-buckets = non-negative-integer *("," non-negative-integer) +nh-f-buckets = float-format-number *("," float-format-number) + +integer = [SIGN] 1*"0" / [SIGN] positive-integer +non-negative-integer = ["+"] 1*"0" / ["+"] positive-integer +; Leading 0s explicitly okay. +positive-integer = *"0" positive-digit *DIGIT +positive-digit = "1" / "2" / "3" / "4" / "5" / "6" / "7" / "8" / "9" + +; float-format-number is a float in the sense that we parse it into a float +; data structure, but may actually have an integer value. +float-format-number = float-format-realnumber +; Case insensitive +float-format-number =/ [SIGN] ("inf" / "infinity") +float-format-number =/ "nan" +; Mush have a dot "." or exponent "e" or both. +; Leading 0s explicitly okay +float-format-realnumber = [SIGN] 1*DIGIT ["." *DIGIT] "e" [SIGN] 1*DIGIT +float-format-realnumber =/ [SIGN] *DIGIT "." 1*DIGIT [ "e" [SIGN] 1*DIGIT ] ``` #### Overall Structure @@ -416,6 +542,16 @@ go_goroutines 69 # UNIT process_cpu_seconds seconds # HELP process_cpu_seconds Total user and system CPU time spent in seconds. process_cpu_seconds_total 4.20072246e+06 +# TYPE acme_http_request_seconds histogram +# UNIT acme_http_request_seconds seconds +# HELP acme_http_request_seconds Latency histogram of all of ACME's HTTP requests. +acme_http_request_seconds{path="/api/v1",method="GET"} {count:2,sum:1.2e2,schema:0,zero_threshold:1e-4,zero_count:0,positive_spans:[1:2],positive_buckets:[1,1]} +acme_http_request_seconds_count{path="/api/v1",method="GET"} 2 +acme_http_request_seconds_sum{path="/api/v1",method="GET"} 1.2e2 +acme_http_request_seconds_buckets{path="/api/v1",method="GET",le="0.5"} 1 +acme_http_request_seconds_buckets{path="/api/v1",method="GET",le="1"} 2 +acme_http_request_seconds_buckets{path="/api/v1",method="GET",le="+Inf"} 2 +acme_http_request_seconds_created{path="/api/v1",method="GET"} 1605281325.0 # EOF ``` @@ -738,14 +874,16 @@ foo{quantile="0.99"} 150.0 Quantiles MAY be in any order. -##### Histogram +##### Histogram with classic buckets -The MetricPoint's Bucket Values Sample MetricNames MUST have the suffix `_bucket`. If present, the MetricPoint's Sum Value Sample MetricName MUST have the suffix `_sum`. If present, the MetricPoint's Created Value Sample MetricName MUST have the suffix `_created`. -If and only if a Sum Value is present in a MetricPoint, then the MetricPoint's +Inf Bucket value MUST also appear in a Sample with a MetricName with the suffix "_count". +The MetricPoint's Classic Bucket Values Sample MetricNames MUST have the suffix `_bucket`. If present, the MetricPoint's Sum Value Sample MetricName MUST have the suffix `_sum`. If present, the MetricPoint's Created Value Sample MetricName MUST have the suffix `_created`. +If and only if a Sum Value is present in a MetricPoint, then the MetricPoint's +Inf Classic Bucket value MUST also appear in a Sample with a MetricName with the suffix "_count". -Buckets MUST be sorted in number increasing order of "le", and the value of the "le" label MUST follow the rules for Canonical Numbers. +Classic Buckets MUST be sorted in number increasing order of "le", and the value of the "le" label MUST follow the rules for Canonical Numbers. + +All Classic Buckets must be represented, even the unpopulated ones that didn't measure any values. -An example of a Metric with no labels and a MetricPoint with Sum, Count, and Created values, and with 12 buckets. A wide and atypical but valid variety of “le” values is shown on purpose: +An example of a Metric with no labels and a MetricPoint with Sum, Count, and Created values, and with 12 classic buckets. A wide and atypical but valid variety of “le” values is shown on purpose: ```openmetrics-add-eof # TYPE foo histogram @@ -765,15 +903,77 @@ foo_sum 324789.3 foo_created 1520430000.123 ``` +##### Histogram with native buckets + +The MetricPoint's value MUST be a complex data type. + +Histograms with native buckets use the integer native histogram data type. + +The integer native histogram data type is a JSON like structure with fields. There MUST NOT be any whitespace around fields. +The integer native histogram data type MUST include the Count, Sum, Schema, Zero Threshold, Zero native bucket value as the fields `count`, `sum`, `schema`, `zero_threshold`, `zero_count`. + +If a positive or negative native bucket is empty, it SHOULD be omitted. See later for a valid use case when to keep empty buckets. + +If there are no negative native buckets, then the fields `negative_spans` and `negative_buckets` SHOULD be omitted. +If there are no positive native buckets, then the fields `positive_spans` and `positive_buckets` SHOULD be omitted. + +If there are negative (and/or positive) native buckets then the fields `negative_spans`, `negative_buckets` (and/or `positive_spans`, `positive_buckets`) MUST be present in this order after the `zero_count` field. + +Native bucket values MUST be ordered by their index, and their values MUST be placed in the `negative_buckets` (and/or `positive_buckets`) fields. + +To map the `negative_buckets` (and/or `positive_buckets`) back to their indices, the `negative_spans` (and/or `positive_spans`) field MUST be constructed in the following way: each span consists of a pair of numbers, an integer called offset and an non-negative integer called length. Only the first span in each list can have a negative offset. It defines the index of the first bucket in its corresponding `negative_buckets` (and/or `positive_buckets`). The length defines the number of consecutive buckets the bucket list starts with. The offsets of the following spans define the number of excluded (and thus unpopulated buckets). The lengths define the number of consecutive buckets in the list following the excluded buckets. + +An example of when to keep empty positive or negative native buckets is to reduce the number of spans needed to represent the case where the offset between two spans is just 1, meaning that with +the inclusion of one empty bucket, the number of spans is reduced by one. + +The sum of all length values in each span list MUST be equal to the length of the corresponding bucket list. + +An example with all fields: + +```openmetrics-add-eof +# TYPE acme_http_request_seconds histogram +acme_http_request_seconds{path="/api/v1",method="GET"} {count:59,sum:1.2e2,schema:7,zero_threshold:1e-4,zero_count:0,negative_spans:[1:2],negative_buckets:[5,7],positive_spans:[-1:2,3:4],positive_buckets:[5,7,10,9,8,8]} +acme_http_request_seconds_created 1520430000.123 +``` + +An example without any buckets in use: + +```openmetrics-add-eof +# TYPE acme_http_request_seconds histogram +acme_http_request_seconds{path="/api/v1",method="GET"} {count:0,sum:0,schema:3,zero_threshold:1e-4,zero_count:0} +acme_http_request_seconds_created 1520430000.123 +``` + +##### Histogram with both classic and native buckets + +If a Histogram MetricPoint has both classic and native buckets, the native buckets MUST come first and the created time MUST NOT be duplicated. + +The order ensures that implementations can easily skip the classic buckets if the native buckets are preferred. + +```openmetrics-add-eof +# TYPE acme_http_request_seconds histogram +# UNIT acme_http_request_seconds seconds +# HELP acme_http_request_seconds Latency histogram of all of ACME's HTTP requests. +acme_http_request_seconds{path="/api/v1",method="GET"} {count:2,sum:1.2e2,schema:0,zero_threshold:1e-4,zero_count:0,positive_spans:[1:2],positive_buckets:[1,1]} +acme_http_request_seconds_count{path="/api/v1",method="GET"} 2 +acme_http_request_seconds_sum{path="/api/v1",method="GET"} 1.2e2 +acme_http_request_seconds_buckets{path="/api/v1",method="GET",le="0.5"} 1 +acme_http_request_seconds_buckets{path="/api/v1",method="GET",le="1"} 2 +acme_http_request_seconds_buckets{path="/api/v1",method="GET",le="+Inf"} 2 +acme_http_request_seconds_created{path="/api/v1",method="GET"} 1605281325.0 +``` + ###### Exemplars Exemplars without Labels MUST represent an empty LabelSet as {}. An example of Exemplars showcasing several valid cases: +The native histogram version of the histogram has multiple Exemplars. The "0.01" bucket has no Exemplar. The 0.1 bucket has an Exemplar with no Labels. The 1 bucket has an Exemplar with one Label. The 10 bucket has an Exemplar with a Label and a timestamp. In practice all buckets SHOULD have the same style of Exemplars. ```openmetrics-add-eof # TYPE foo histogram +foo {count:10,sum:1.0,schema:0,zero_threshold:1e-4,zero_count:0,positive_spans:[0:2],positive_buckets:[5,5]} # {trace_id="shaZ8oxi"} 0.67 1520879607.789 # {trace_id="ookahn0M"} 1.2 1520879608.589 foo_bucket{le="0.01"} 0 foo_bucket{le="0.1"} 8 # {} 0.054 foo_bucket{le="1"} 11 # {trace_id="KOO5S4vxi0o"} 0.67 @@ -784,7 +984,7 @@ foo_sum 324789.3 foo_created 1520430000.123 ``` -##### GaugeHistogram +##### GaugeHistogram with classic buckets The MetricPoint's Bucket Values Sample MetricNames MUST have the suffix `_bucket`. If present, the MetricPoint's Sum Value Sample MetricName MUST have the suffix `_gsum`. If and only if a Sum Value is present in a MetricPoint, then the MetricPoint's +Inf Bucket value MUST also appear in a Sample with a MetricName with the suffix `_gcount`. @@ -804,6 +1004,20 @@ foo_gcount 42.0 foo_gsum 3289.3 ``` +##### GaugeHistogram with native buckets + +GaugeHistogram MetricPoints with native buckets follow the same syntax as Histogram MetricPoints with native buckets. + +```openmetrics-add-eof +# TYPE acme_http_request_seconds gaugehistogram +acme_http_request_seconds{path="/api/v1",method="GET"} {count:59,sum:1.2e2,schema:7,zero_threshold:1e-4,zero_count:0,negative_spans:[1:2],negative_buckets:[5,7],positive_spans:[-1:2,3:4],positive_buckets:[5,7,10,9,8,8]} +acme_http_request_seconds_created 1520430000.123 +``` + +##### GaugeHistogram with both classic and native buckets + +If a GaugeHistogram MetricPoint has both classic and native buckets, the native buckets MUST come first and the created time MUST NOT be duplicated. + ##### Unknown The sample metric name for the value of the MetricPoint for a MetricFamily of type Unknown MUST NOT have a suffix.