You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/main/_sources/quantiles/index.rst.txt
+9-5Lines changed: 9 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,17 +10,21 @@ in the stream.
10
10
These sketches may be used to compute approximate histograms, Probability Mass Functions (PMFs), or
11
11
Cumulative Distribution Functions (CDFs).
12
12
13
-
The library provides three types of quantiles sketches, each of which has generic items as well as versions
14
-
specific to a given numeric type (e.g. integer or floating point values). All three types provide error
15
-
bounds on rank estimation with proven probabilistic error distributions.
13
+
The library provides four types of quantiles sketches, three of which have generic items as well as versions
14
+
specific to a given numeric type (e.g. integer or floating point values). Those three types provide error
15
+
bounds on rank estimation with proven probabilistic error distributions. t-digest is a heuristic-based sketch
16
+
that works only on numeric data, and while the error properties are not guaranteed, the sketch typically
17
+
does a good job with small storage.
16
18
17
-
* KLL: Provides uniform rank estimation error over the entire range
19
+
* KLL: Provides uniform rank estimation error over the entire range.
18
20
* REQ: Provides relative rank error estimates, which decreases approaching either the high or low end values.
21
+
* t-digest: Relative rank error estimates, heuristic-based without guarantees but quite compact with generally very good error properties.
19
22
* Classic quantiles: Largely deprecated in favor of KLL, also provides uniform rank estimation error. Included largely for backwards compatibility with historic data.
The implementation in this library is based on the MergingDigest described in
7
+
`Computing Extremely Accurate Quantiles Using t-Digests <https://arxiv.org/abs/1902.04023>`_ by Ted Dunning and Otmar Ertl.
8
+
9
+
The implementation in this library has a few differences from the reference implementation associated with that paper:
10
+
11
+
* Merge does not modify the input
12
+
* Derialization similar to other sketches in this library, although reading the reference implementation format is supported
13
+
14
+
Unlike all other algorithms in the library, t-digest is empirical and has no mathematical basis for estimating its error
15
+
and its results are dependent on the input data. However, for many common data distributions, it can produce excellent results.
16
+
t-digest also operates only on numeric data and, unlike the quantiles family algorithms in the library which return quantile
17
+
approximations from the input domain, t-digest interpolates values and will hold and return data points not seen in the input.
18
+
19
+
The closest alternative to t-digest in this library is REQ sketch. It prioritizes one chosen side of the rank domain:
20
+
either low rank accuracy or high rank accuracy. t-digest (in this implementation) prioritizes both ends of the rank domain
21
+
and has lower accuracy towards the middle of the rank domain (median).
22
+
23
+
Measurements show that t-digest is slightly biased (tends to underestimate low ranks and overestimate high ranks), while still
24
+
doing very well close to the extremes. The effect seems to be more pronounced with more input values.
25
+
26
+
For more information on the performance characteristics, see `the Datasketches page on t-digest <https://datasketches.apache.org/docs/tdigest/tdigest.html>`_.
0 commit comments