## Class EmpiricalDistribution

• All Implemented Interfaces:
org.apache.commons.statistics.distribution.ContinuousDistribution

public final class EmpiricalDistribution
extends AbstractRealDistribution
implements org.apache.commons.statistics.distribution.ContinuousDistribution

Represents an empirical probability distribution: Probability distribution derived from observed data without making any assumptions about the functional form of the population distribution that the data come from.

An EmpiricalDistribution maintains data structures called distribution digests that describe empirical distributions and support the following operations:

• dividing the input data into "bin ranges" and reporting bin frequency counts (data for histogram)
• reporting univariate statistics describing the full set of data values as well as the observations within each bin
• generating random values from the distribution
Applications can use EmpiricalDistribution to build grouped frequency histograms representing the input data or to generate random values "like" those in the input, i.e. the values generated will follow the distribution of the values in the file.

The implementation uses what amounts to the Variable Kernel Method with Gaussian smoothing:

Digesting the input file

1. Pass the file once to compute min and max.
2. Divide the range from min to max into binCount bins.
3. Pass the data file again, computing bin counts and univariate statistics (mean and std dev.) for each bin.
4. Divide the interval (0,1) into subintervals associated with the bins, with the length of a bin's subinterval proportional to its count.
Generating random values from the distribution
1. Generate a uniformly distributed value in (0,1)
2. Select the subinterval to which the value belongs.
3. Generate a random Gaussian value with mean = mean of the associated bin and std dev = std dev of associated bin.

EmpiricalDistribution implements the ContinuousDistribution interface as follows. Given x within the range of values in the dataset, let B be the bin containing x and let K be the within-bin kernel for B. Let P(B-) be the sum of the probabilities of the bins below B and let K(B) be the mass of B under K (i.e., the integral of the kernel density over B). Then set P(X < x) = P(B-) + P(B) * K(x) / K(B) where K(x) is the kernel distribution evaluated at x. This results in a cdf that matches the grouped frequency distribution at the bin endpoints and interpolates within bins using within-bin kernels.

CAVEAT: It is advised that the bin count is about one tenth of the size of the input array.

• ### Nested classes/interfaces inherited from interface org.apache.commons.statistics.distribution.ContinuousDistribution

org.apache.commons.statistics.distribution.ContinuousDistribution.Sampler

• ### Fields inherited from class org.apache.commons.math4.legacy.distribution.AbstractRealDistribution

SOLVER_DEFAULT_ABSOLUTE_ACCURACY
• ### Method Summary

All Methods
Modifier and Type Method Description
double cumulativeProbability​(double x)
Algorithm description: Find the bin B that x belongs to. Compute P(B) = the mass of B and P(B-) = the combined mass of the bins below B. Compute K(B) = the probability mass of B with respect to the within-bin kernel and K(B-) = the kernel distribution evaluated at the lower endpoint of B Return P(B-) + P(B) * [K(x) - K(B-)] / K(B) where K(x) is the within-bin kernel distribution function evaluated at x. If K is a constant distribution, we return P(B-) + P(B) (counting the full mass of B).
double density​(double x)
Returns the kernel density normalized so that its integral over each bin equals the bin mass.
static EmpiricalDistribution from​(int binCount, double[] input)
Factory that creates a new instance from the specified data.
static EmpiricalDistribution from​(int binCount, double[] input, Function<SummaryStatistics,​org.apache.commons.statistics.distribution.ContinuousDistribution> kernelFactory)
Factory that creates a new instance from the specified data.
int getBinCount()
Returns the number of bins.
List<SummaryStatistics> getBinStats()
Returns a copy of the SummaryStatistics instances containing statistics describing the values in each of the bins.
double[] getGeneratorUpperBounds()
Returns the upper bounds of the subintervals of [0, 1] used in generating data from the empirical distribution.
double getMean()
StatisticalSummary getSampleStats()
Returns a StatisticalSummary describing this distribution.
double getSupportLowerBound()
double getSupportUpperBound()
double[] getUpperBounds()
Returns the upper bounds of the bins.
double getVariance()
double inverseCumulativeProbability​(double p)
The default implementation returns ContinuousDistribution.getSupportLowerBound() for p = 0, ContinuousDistribution.getSupportUpperBound() for p = 1.
• ### Methods inherited from class org.apache.commons.math4.legacy.distribution.AbstractRealDistribution

createSampler, getSolverAbsoluteAccuracy, logDensity, probability, sample
• ### Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
• ### Methods inherited from interface org.apache.commons.statistics.distribution.ContinuousDistribution

createSampler, inverseSurvivalProbability, logDensity, probability, survivalProbability
• ### Method Detail

• #### from

public static EmpiricalDistribution from​(int binCount,
double[] input,
Function<SummaryStatistics,​org.apache.commons.statistics.distribution.ContinuousDistribution> kernelFactory)
Factory that creates a new instance from the specified data.
Parameters:
binCount - Number of bins. Must be strictly positive.
input - Input data. Cannot be null.
kernelFactory - Factory for creating within-bin kernels.
Returns:
a new instance.
Throws:
NotStrictlyPositiveException - if binCount <= 0.
• #### from

public static EmpiricalDistribution from​(int binCount,
double[] input)
Factory that creates a new instance from the specified data.
Parameters:
binCount - Number of bins. Must be strictly positive.
input - Input data. Cannot be null.
Returns:
a new instance.
Throws:
NotStrictlyPositiveException - if binCount <= 0.
• #### getSampleStats

public StatisticalSummary getSampleStats()
Returns a StatisticalSummary describing this distribution. Preconditions:
• the distribution must be loaded before invoking this method
Returns:
the sample statistics
Throws:
IllegalStateException - if the distribution has not been loaded
• #### getBinCount

public int getBinCount()
Returns the number of bins.
Returns:
the number of bins.
• #### getBinStats

public List<SummaryStatistics> getBinStats()
Returns a copy of the SummaryStatistics instances containing statistics describing the values in each of the bins. The list is indexed on the bin number.
Returns:
the bins statistics.
• #### getUpperBounds

public double[] getUpperBounds()
Returns the upper bounds of the bins. Assuming array u is returned by this method, the bins are:
• (min, u[0]),
• (u[0], u[1]),
• ... ,
• (u[binCount - 2], u[binCount - 1] = max),
Returns:
the bins upper bounds.
Since:
2.1
• #### getGeneratorUpperBounds

public double[] getGeneratorUpperBounds()
Returns the upper bounds of the subintervals of [0, 1] used in generating data from the empirical distribution. Subintervals correspond to bins with lengths proportional to bin counts. Preconditions:
• the distribution must be loaded before invoking this method
Returns:
array of upper bounds of subintervals used in data generation
Throws:
NullPointerException - unless a load method has been called beforehand.
Since:
2.1
• #### density

public double density​(double x)
Returns the kernel density normalized so that its integral over each bin equals the bin mass. Algorithm description:
1. Find the bin B that x belongs to.
2. Compute K(B) = the mass of B with respect to the within-bin kernel (i.e., the integral of the kernel density over B).
3. Return k(x) * P(B) / K(B), where k is the within-bin kernel density and P(B) is the mass of B.
Specified by:
density in interface org.apache.commons.statistics.distribution.ContinuousDistribution
Since:
3.1
• #### cumulativeProbability

public double cumulativeProbability​(double x)
Algorithm description:
1. Find the bin B that x belongs to.
2. Compute P(B) = the mass of B and P(B-) = the combined mass of the bins below B.
3. Compute K(B) = the probability mass of B with respect to the within-bin kernel and K(B-) = the kernel distribution evaluated at the lower endpoint of B
4. Return P(B-) + P(B) * [K(x) - K(B-)] / K(B) where K(x) is the within-bin kernel distribution function evaluated at x.
If K is a constant distribution, we return P(B-) + P(B) (counting the full mass of B).
Specified by:
cumulativeProbability in interface org.apache.commons.statistics.distribution.ContinuousDistribution
Since:
3.1
• #### inverseCumulativeProbability

public double inverseCumulativeProbability​(double p)
The default implementation returns
• ContinuousDistribution.getSupportLowerBound() for p = 0,
• ContinuousDistribution.getSupportUpperBound() for p = 1.
Algorithm description:
1. Find the smallest i such that the sum of the masses of the bins through i is at least p.
1. Let K be the within-bin kernel distribution for bin i.
2. Let K(B) be the mass of B under K.
3. Let K(B-) be K evaluated at the lower endpoint of B (the combined mass of the bins below B under K).
4. Let P(B) be the probability of bin i.
5. Let P(B-) be the sum of the bin masses below bin i.
6. Let pCrit = p - P(B-)
2. Return the inverse of K evaluated at K(B-) + pCrit * K(B) / P(B)
Specified by:
inverseCumulativeProbability in interface org.apache.commons.statistics.distribution.ContinuousDistribution
Overrides:
inverseCumulativeProbability in class AbstractRealDistribution
Since:
3.1
• #### getMean

public double getMean()
Specified by:
getMean in interface org.apache.commons.statistics.distribution.ContinuousDistribution
Since:
3.1
• #### getVariance

public double getVariance()
Specified by:
getVariance in interface org.apache.commons.statistics.distribution.ContinuousDistribution
Since:
3.1
• #### getSupportLowerBound

public double getSupportLowerBound()
Specified by:
getSupportLowerBound in interface org.apache.commons.statistics.distribution.ContinuousDistribution
Since:
3.1
• #### getSupportUpperBound

public double getSupportUpperBound()
Specified by:
getSupportUpperBound in interface org.apache.commons.statistics.distribution.ContinuousDistribution
Since:
3.1