Package org.apache.commons.math4.legacy.distribution

Class EmpiricalDistribution

  • All Implemented Interfaces:
    org.apache.commons.statistics.distribution.ContinuousDistribution
    public final class EmpiricalDistribution
extends AbstractRealDistribution
implements org.apache.commons.statistics.distribution.ContinuousDistribution

    Represents an empirical probability distribution: Probability distribution derived from observed data without making any assumptions about the functional form of the population distribution that the data come from.

    An EmpiricalDistribution maintains data structures called distribution digests that describe empirical distributions and support the following operations:

    • loading the distribution from "observed" data values
    • dividing the input data into "bin ranges" and reporting bin frequency counts (data for histogram)
    • reporting univariate statistics describing the full set of data values as well as the observations within each bin
    • generating random values from the distribution
    Applications can use EmpiricalDistribution to build grouped frequency histograms representing the input data or to generate random values "like" those in the input, i.e. the values generated will follow the distribution of the values in the file.

    The implementation uses what amounts to the Variable Kernel Method with Gaussian smoothing:

    Digesting the input file

    1. Pass the file once to compute min and max.
    2. Divide the range from min to max into binCount bins.
    3. Pass the data file again, computing bin counts and univariate statistics (mean and std dev.) for each bin.
    4. Divide the interval (0,1) into subintervals associated with the bins, with the length of a bin's subinterval proportional to its count.
    Generating random values from the distribution
    1. Generate a uniformly distributed value in (0,1)
    2. Select the subinterval to which the value belongs.
    3. Generate a random Gaussian value with mean = mean of the associated bin and std dev = std dev of associated bin.

    EmpiricalDistribution implements the ContinuousDistribution interface as follows. Given x within the range of values in the dataset, let B be the bin containing x and let K be the within-bin kernel for B. Let P(B-) be the sum of the probabilities of the bins below B and let K(B) be the mass of B under K (i.e., the integral of the kernel density over B). Then set P(X < x) = P(B-) + P(B) * K(x) / K(B) where K(x) is the kernel distribution evaluated at x. This results in a cdf that matches the grouped frequency distribution at the bin endpoints and interpolates within bins using within-bin kernels.

    CAVEAT: It is advised that the bin count is about one tenth of the size of the input array.

    • Method Detail

      • from

        public static EmpiricalDistribution from​(int binCount,
                                         double[] input,
                                         Function<SummaryStatistics,​org.apache.commons.statistics.distribution.ContinuousDistribution> kernelFactory)
        Factory that creates a new instance from the specified data.
        Parameters:
        binCount - Number of bins. Must be strictly positive.
        input - Input data. Cannot be null.
        kernelFactory - Factory for creating within-bin kernels.
        Returns:
        a new instance.
        Throws:
        NotStrictlyPositiveException - if binCount <= 0.

      • from

        public static EmpiricalDistribution from​(int binCount,
                                         double[] input)
        Factory that creates a new instance from the specified data.
        Parameters:
        binCount - Number of bins. Must be strictly positive.
        input - Input data. Cannot be null.
        Returns:
        a new instance.
        Throws:
        NotStrictlyPositiveException - if binCount <= 0.

      • getBinCount

        public int getBinCount()
        Returns the number of bins.
        Returns:
        the number of bins.

      • getUpperBounds

        public double[] getUpperBounds()
        Returns the upper bounds of the bins. Assuming array u is returned by this method, the bins are:
        • (min, u[0]),
        • (u[0], u[1]),
        • ... ,
        • (u[binCount - 2], u[binCount - 1] = max),
        Returns:
        the bins upper bounds.
        Since:
        2.1

      • getGeneratorUpperBounds

        public double[] getGeneratorUpperBounds()
        Returns the upper bounds of the subintervals of [0, 1] used in generating data from the empirical distribution. Subintervals correspond to bins with lengths proportional to bin counts. Preconditions:
        • the distribution must be loaded before invoking this method
        Returns:
        array of upper bounds of subintervals used in data generation
        Throws:
        NullPointerException - unless a load method has been called beforehand.
        Since:
        2.1

      • density

        public double density​(double x)
        Returns the kernel density normalized so that its integral over each bin equals the bin mass. Algorithm description:
        1. Find the bin B that x belongs to.
        2. Compute K(B) = the mass of B with respect to the within-bin kernel (i.e., the integral of the kernel density over B).
        3. Return k(x) * P(B) / K(B), where k is the within-bin kernel density and P(B) is the mass of B.
        Specified by:
        density in interface org.apache.commons.statistics.distribution.ContinuousDistribution
        Since:
        3.1

      • cumulativeProbability

        public double cumulativeProbability​(double x)
        Algorithm description:
        1. Find the bin B that x belongs to.
        2. Compute P(B) = the mass of B and P(B-) = the combined mass of the bins below B.
        3. Compute K(B) = the probability mass of B with respect to the within-bin kernel and K(B-) = the kernel distribution evaluated at the lower endpoint of B
        4. Return P(B-) + P(B) * [K(x) - K(B-)] / K(B) where K(x) is the within-bin kernel distribution function evaluated at x.
        If K is a constant distribution, we return P(B-) + P(B) (counting the full mass of B).
        Specified by:
        cumulativeProbability in interface org.apache.commons.statistics.distribution.ContinuousDistribution
        Since:
        3.1

      • inverseCumulativeProbability

        public double inverseCumulativeProbability​(double p)
        The default implementation returns
        • ContinuousDistribution.getSupportLowerBound() for p = 0,
        • ContinuousDistribution.getSupportUpperBound() for p = 1.
        Algorithm description:
        1. Find the smallest i such that the sum of the masses of the bins through i is at least p.
          1. Let K be the within-bin kernel distribution for bin i.
          2. Let K(B) be the mass of B under K.
          3. Let K(B-) be K evaluated at the lower endpoint of B (the combined mass of the bins below B under K).
          4. Let P(B) be the probability of bin i.
          5. Let P(B-) be the sum of the bin masses below bin i.
          6. Let pCrit = p - P(B-)
        3. Return the inverse of K evaluated at K(B-) + pCrit * K(B) / P(B)
        Specified by:
        inverseCumulativeProbability in interface org.apache.commons.statistics.distribution.ContinuousDistribution
        Overrides:
        inverseCumulativeProbability in class AbstractRealDistribution
        Since:
        3.1

      • getMean

        public double getMean()
        Specified by:
        getMean in interface org.apache.commons.statistics.distribution.ContinuousDistribution
        Since:
        3.1

      • getVariance

        public double getVariance()
        Specified by:
        getVariance in interface org.apache.commons.statistics.distribution.ContinuousDistribution
        Since:
        3.1

      • getSupportLowerBound

        public double getSupportLowerBound()
        Specified by:
        getSupportLowerBound in interface org.apache.commons.statistics.distribution.ContinuousDistribution
        Since:
        3.1

      • getSupportUpperBound

        public double getSupportUpperBound()
        Specified by:
        getSupportUpperBound in interface org.apache.commons.statistics.distribution.ContinuousDistribution
        Since:
        3.1