The Normal (Gaussian) distribution is the centerpiece of continuous probability for ML, with PDF \(f(x) = \frac{1}{\sigma\sqrt{2\pi}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)\), parameterized by mean \(\mu\) and variance \(\sigma^2\). The standard Normal with \(\mu = 0\) and \(\sigma^2 = 1\) has CDF denoted \(\Phi\) and PDF \(\phi(z) = \frac{1}{\sqrt{2\pi}} e^{-z^2/2}\). A useful heuristic is the 68-95-99.7 rule: roughly 68% of the probability mass lies within 1\(\sigma\) of the mean, 95% within 2\(\sigma\), and 99.7% within 3\(\sigma\). The Central Limit Theorem (CLT) explains the Normals ubiquity: the sum (properly normalized) of \(n\) i.i.d. random variables with finite mean \(\mu\) and variance \(\sigma^2\) converges in distribution to Normal(\(\mu, \sigma^2/n\)) as \(n\) grows. This justifies treating averages, losses, and estimator errors as approximately Normal in machine learning, enabling confidence intervals, z-tests, and Gaussian approximations of aggregate behavior.
Sampling from a standard Normal is often done via the Box–Muller transform, which maps two independent U(0, 1) uniforms to two independent standard Normals: \[Z_0 = \sqrt{-2 \ln U_1} \cos(2\pi U_2), \quad Z_1 = \sqrt{-2 \ln U_1} \sin(2\pi U_2).\] The multivariate extension, the multivariate Gaussian, places a \(d\)-dimensional Normal with density \(f(x) = (2\pi)^{-d/2} |\Sigma|^{-1/2} \exp\left(-\tfrac{1}{2}(x - \mu)^T \Sigma^{-1} (x - \mu)\right)\), where \(\Sigma\) must be positive definite. Distances in this space are measured by the Mahalanobis distance \(d(x, \mu) = \sqrt{(x - \mu)^T \Sigma^{-1} (x - \mu)}\), which scales deviations by the covariance structure. An isotropic Gaussian has \(\Sigma = \sigma^2 I_d\), making its density depend only on Euclidean distance and giving it rotational symmetry. A singular (or degenerate) Gaussian has a singular \(\Sigma\), so its mass lies on a lower-dimensional affine subspace and the density is undefined in the ambient space. Whitening applies a linear transformation \(z = L^{-1}(x - \mu)\), where \(L\) is typically the Cholesky factor of \(\Sigma\) with \(LL^T = \Sigma\), mapping a centered multivariate Normal to a standard Normal.
Several matrix-valued distributions extend the Gaussian framework to covariance estimation and Bayesian modeling. The Wishart distribution generalizes chi-squared to matrices, governing the distribution of sums of outer products of multivariate Gaussian vectors; it serves as a conjugate prior for the inverse covariance of a multivariate Normal. The inverse-Wishart plays the conjugate role for the covariance \(\Sigma\) itself: with \(\Sigma \sim\) Inv-Wishart(\(\Psi, \nu\)) and i.i.d. multivariate Normal data, the posterior is also inverse-Wishart. The LKJ prior is a distribution over correlation matrices indexed by a shape parameter \(\eta \ge 0\), with \(\eta = 1\) producing a uniform distribution over correlation matrices and larger \(\eta\) concentrating mass near the identity. Finally, the reparameterization trick — writing \(x = \mu + \sigma \odot \varepsilon\) with \(\varepsilon \sim N(0, I)\) — allows gradients of expectations \(E_{x \sim N(\mu, \sigma^2)}[f(x)]\) to flow through \(\mu\) and \(\sigma\), and is foundational to variational autoencoders and other gradient-based generative models.