Skip to content

Chapter 2 of 8

Discrete Distributions for ML

A family of closely related discrete distributions arises from independent Bernoulli trials. The Bernoulli distribution itself models a single binary outcome: \(P(X = 1) = p\), \(P(X = 0) = 1 - p\), with mean \(p\) and variance \(p(1 - p)\). The Binomial distribution counts the number of successes across \(n\) independent Bernoulli(\(p\)) trials, with PMF \(P(X = k) = \binom{n}{k} p^k (1-p)^{n-k}\) for \(k = 0, 1, \dots, n\), mean \(np\), and variance \(np(1-p)\). Generalizing from two outcomes to \(K\) categories gives the Categorical (or Multinoulli) distribution, which samples a single category with probabilities summing to one, and the Multinomial distribution, which counts outcomes across \(K\) categories in \(n\) independent draws with joint PMF \(\frac{n!}{\prod_i x_i!} \prod_i p_i^{x_i}\).

Several other discrete distributions handle waiting-time and event-counting problems. The Geometric distribution (in its "number-of-trials" form) gives the trial index of the first success, \(P(X = k) = (1-p)^{k-1} p\) for \(k = 1, 2, \dots\), with mean \(1/p\); the equivalent "failures-before-success" form is \(P(X = k) = (1-p)^k p\) starting at \(k = 0\), with mean \((1-p)/p\). Extending this idea, the Negative Binomial distribution counts failures before the \(r\)-th success, with PMF \(P(X = k) = \binom{r+k-1}{k} p^r (1-p)^k\), mean \(r(1-p)/p\), and variance \(r(1-p)/p^2\). The Poisson distribution models counts of independent events at constant rate \(\lambda\) over a fixed interval, \(P(X = k) = e^{-\lambda} \lambda^k / k!\), with both mean and variance equal to \(\lambda\). It is a good model for rare events such as decay events, web page hits per minute, or arrival counts.

The Poisson distribution emerges naturally as a limit of the Binomial: when \(p \to 0\) and \(n \to \infty\) with \(np = \lambda\) held fixed, \(P(X = k) \approx e^{-\lambda} \lambda^k / k!\). This justifies using Poisson as the "rare events" approximation. As a useful implementation detail, \(k\) independent Bernoulli(0.5) samples can be combined into a uniformly distributed integer in \(\{0, \dots, 2^k - 1\}\) by interpreting the bit string as binary — an example of how simple distributions compose to produce more complex ones.

All chapters
  1. 1Foundations of Probability Distributions
  2. 2Discrete Distributions for ML
  3. 3Continuous Distributions
  4. 4The Normal Family and Asymptotics
  5. 5Bayesian Inference and Conjugate Priors
  6. 6Mixtures, Latent Variables, and Variational Bounds
  7. 7Classification Models from a Distributional View
  8. 8Sampling, MCMC, and Modern Generative Models

Drill it

Reading is not remembering. These come from the Probability Distributions For Ml deck:

Q

What is a probability distribution?

A function (or rule) that assigns to every outcome in a sample space a non-negative probability, with the total probability over all outcomes summing (or integr...

Q

What is a discrete probability distribution?

A distribution defined on a countable set of outcomes, specified by a probability mass function (PMF) p(x) with p(x) ≥ 0 and Σ p(x) = 1.

Q

What is a probability mass function (PMF)?

A function p(x) that gives the probability of a discrete random variable taking the value x; satisfies p(x) ≥ 0 for all x and Σ_x p(x) = 1.

Q

What is a probability density function (PDF)?

A function f(x) for a continuous random variable such that P(a ≤ X ≤ b) = ∫_a^b f(x) dx, with f(x) ≥ 0 and ∫_{-∞}^{∞} f(x) dx = 1.