public interface QuantilesAPI
This is a stochastic streaming sketch that enables near-real time analysis of the approximate distribution of items from a very large stream in a single pass, requiring only that the items are comparable. The analysis is obtained using the getQuantile() function or the inverse functions getRank(), getPMF() (the Probability Mass Function), and getCDF() (the Cumulative Distribution Function).
Given an input stream of N items, the natural rank of any specific item is defined as its index (1 to N) in the hypothetical sorted stream of all N input items.
The normalized rank (rank) of any specific item is defined as its natural rank divided by N, which is a number in the interval [0.0, 1.0]. In the Javadocs for all the quantile sketches natural rank is seldom used so any reference to just rank should be interpreted as normalized rank.
Inputs into a quantile sketch are called "items" that can be either generic or specific primitives, like float or double depending on the sketch implementation. In order to keep its size small, sketches don't retain all the items offered and retain only a small fraction of all the items, thus purging most of the items. The items retained are then sorted and associated with a rank. At this point we call the retained items quantiles. Thus, all quantiles are items, but only a few items become quantiles. Depending on the context the two terms can be interchangeable.
All quantile sketches are configured with a parameter k, which affects the size of the sketch and its estimation error.
In the research literature, the estimation error is commonly called epsilon (or eps) and is a fraction between zero and one. Larger sizes of k result in a smaller epsilon, but also a larger sketch. The epsilon error is always with respect to the rank domain. Estimating the confidence interval in the quantile domain can be done by first computing the error in the rank domain and then translating that to the quantile domain. The sketch provides methods to assist with that.
The relationship between the normalized rank and the corresponding quantiles can be viewed as a two dimensional monotonic plot with the normalized rank on one axis and the corresponding quantiles on the other axis. Let q := quantile and r := rank then both q = getQuantile(r) and r = getRank(q) are monotonically increasing functions. If the y-axis is used for the rank domain and the x-axis for the quantile domain, then y = getRank(x) is also the single point Cumulative Distribution Function (CDF).
The functions getQuantile() translate ranks into corresponding quantiles. The functions getRank(), getCDF(), and getPMF() (Probability Mass Function) perform the opposite operation and translate quantiles into ranks (or cumulative probabilities, or probability masses, depending on the context).
As an example, consider a large stream of one million items such as packet sizes coming into a network node. The absolute rank of any specific item size is simply its index in the hypothetical sorted array of such items. The normalized rank is the natural rank divided by the stream size, or N, in this case one million. The quantile corresponding to the normalized rank of 0.5 represents the 50th percentile or median of the distribution, obtained from getQuantile(0.5). Similarly, the 95th percentile is obtained from getQuantile(0.95).
From the min and max quantiles, for example, say 1 and 1000 bytes, you can obtain the PMF from getPMF(100, 500, 900) that will result in an array of 4 probability masses such as {.4, .3, .2, .1}, which means that
The accuracy of this sketch is a function of the configured k, which also affects the overall size of the sketch. Accuracy of this quantile sketch is always with respect to the normalized rank.
The getPMF() function has about 13 to 47% worse rank error (depending on k) than the other queries because the mass of each "bin" of the PMF has "double-sided" error from the upper and lower edges of the bin as a result of a subtraction of random variables where the errors from the two edges can sometimes add.
A getQuantile(rank) query has the following probabilistic guarantees:
A getRank(quantile) query has the following probabilistic guarantees:
A getPMF() query has the following probabilistic guarantees:
A getCDF(...) query has the following probabilistic guarantees:
Because errors are independent, we can make some estimates of the size of the confidence bounds for the quantile returned from a call to getQuantile(), but not error bounds. These confidence bounds may be quite large for certain distributions.
This sketch is order and distribution insensitive
This algorithm intentionally inserts randomness into the sampling process for items that ultimately get retained in the sketch. Thus, the results produced by this algorithm are not deterministic. For example, if the same stream is inserted into two different instances of this sketch, the answers obtained from the two sketches should be close, but may not be be identical.
Similarly, there may be directional inconsistencies. For example, if a quantile obtained from getQuantile(rank) is input into the reverse query getRank(quantile), the resulting rank should be close, but may not exactly equal the original rank.
Please visit our website: DataSketches Home Page and specific Javadocs for more information.
[*] Note that obtaining epsilon may require using a similar function but with more parameters based on the specific sketch implementation.
QuantileSearchCriteria
Modifier and Type | Field and Description |
---|---|
static String |
EMPTY_MSG |
static String |
MEM_REQ_SVR_NULL_MSG |
static String |
NOT_SINGLE_ITEM_MSG |
static String |
SELF_MERGE_MSG |
static String |
TGT_IS_READ_ONLY_MSG |
static String |
UNSUPPORTED_MSG |
Modifier and Type | Method and Description |
---|---|
int |
getK()
Gets the user configured parameter k, which controls the accuracy of the sketch
and its memory space usage.
|
long |
getN()
Gets the length of the input stream offered to the sketch..
|
double |
getNormalizedRankError(boolean pmf)
Gets the approximate rank error of this sketch normalized as a fraction between zero and one.
|
int |
getNumRetained()
Gets the number of quantiles retained by the sketch.
|
double |
getRankLowerBound(double rank)
Gets the lower bound of the rank confidence interval in which the true rank of the
given rank exists.
|
double |
getRankUpperBound(double rank)
Gets the upper bound of the rank confidence interval in which the true rank of the
given rank exists.
|
boolean |
hasMemory()
Returns true if this sketch's data structure is backed by Memory or WritableMemory.
|
boolean |
isDirect()
Returns true if this sketch's data structure is off-heap (a.k.a., Direct or Native memory).
|
boolean |
isEmpty()
Returns true if this sketch is empty.
|
boolean |
isEstimationMode()
Returns true if this sketch is in estimation mode.
|
boolean |
isReadOnly()
Returns true if this sketch is read only.
|
void |
reset()
Resets this sketch to the empty state.
|
String |
toString()
Returns a summary of the key parameters of the sketch.
|
static final String EMPTY_MSG
static final String UNSUPPORTED_MSG
static final String NOT_SINGLE_ITEM_MSG
static final String MEM_REQ_SVR_NULL_MSG
static final String TGT_IS_READ_ONLY_MSG
static final String SELF_MERGE_MSG
int getK()
long getN()
double getNormalizedRankError(boolean pmf)
pmf
- if true, returns the "double-sided" normalized rank error for the getPMF() function.
Otherwise, it is the "single-sided" normalized rank error for all the other queries.int getNumRetained()
double getRankLowerBound(double rank)
rank
- the given normalized rank.double getRankUpperBound(double rank)
rank
- the given normalized rank.boolean hasMemory()
boolean isDirect()
boolean isEmpty()
boolean isEstimationMode()
boolean isReadOnly()
void reset()
The parameter k will not change.
Copyright © 2015–2024 The Apache Software Foundation. All rights reserved.