Probability axioms form the foundation. Kolmogorov axioms: \(P(\Omega) = 1\), \(P(A) \geq 0\), and \(P(A \cup B) = P(A) + P(B)\) for disjoint \(A, B\). The complement rule \(P(A^c) = 1 - P(A)\) and inclusion-exclusion \(P(A \cup B) = P(A) + P(B) - P(A \cap B)\) extend these. Conditional probability \(P(A|B) = P(A \cap B)/P(B)\) restricts the sample space; independence \(P(A \cap B) = P(A)P(B)\) means \(B\) tells nothing about \(A\). The law of total probability marginalizes: \(P(A) = \sum_i P(A|B_i)P(B_i)\). Bayes' theorem \(P(\theta|x) = P(x|\theta)P(\theta)/P(x)\) updates prior beliefs: posterior \(\propto\) likelihood \(\times\) prior. The evidence (marginal likelihood) \(P(x) = \int P(x|\theta)P(\theta)d\theta\) is often intractable—requiring variational inference or MCMC. With i.i.d. observations, the Bayesian update becomes \(P(\theta|x_1,\ldots,x_n) \propto (\prod_i P(x_i|\theta)) P(\theta)\). Conditional independence \(X \perp Y|Z\) means \(P(X,Y|Z) = P(X|Z)P(Y|Z)\)—fundamental in graphical models and Bayesian networks. The Markov property \(P(X_{t+1}|X_t, X_{t-1},\ldots,X_1) = P(X_{t+1}|X_t)\) states the future is independent of the past given the present.
Key distributions cover discrete and continuous cases. Bernoulli: \(P(X=1) = p\), \(P(X=0) = 1-p\), max variance at \(p=0.5\)—foundation for binary classification. Binomial: sum of n Bernoulli trials with \(E[X] = np\), \(\mathrm{Var}(X) = np(1-p)\), approaches Gaussian for large \(n\). Poisson: \(P(X=k) = \lambda^k e^{-\lambda}/k!\) with mean and variance both equal \(\lambda\)—models rare events like word counts in NLP. Geometric: number of trials to first success, memoryless \(P(X>m+n|X>m) = P(X>n)\). Gaussian (Normal): \(\mathcal{N}(x|\mu,\sigma^2) = (1/\sqrt{2\pi\sigma^2})\exp(-(x-\mu)^2/(2\sigma^2))\)—the most important distribution, with maximum entropy under fixed mean/variance and closed under linear operations. The 68-95-99.7 rule gives standard-deviation-based probability mass. The standard normal \(Z = (X-\mu)/\sigma \sim \mathcal{N}(0,1)\) normalizes any Gaussian. Multivariate Gaussian: \(\mathcal{N}(\mathbf{x}|\boldsymbol{\mu},\Sigma) = (2\pi)^{-d/2}|\Sigma|^{-1/2}\exp(-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu})^\top\Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu}))\), with the squared Mahalanobis distance \(d_M(\mathbf{x},\boldsymbol{\mu})^2 = (\mathbf{x}-\boldsymbol{\mu})^\top \Sigma^{-1} (\mathbf{x}-\boldsymbol{\mu})\) measuring deviations. Categorical extends Bernoulli to K outcomes; Dirichlet is the conjugate prior over probability vectors; Beta is conjugate to Bernoulli/Binomial with \(\mathbb{E}[p] = \alpha/(\alpha+\beta)\). A diagonal covariance matrix implies uncorrelated features. The entropy of a Gaussian is \(\frac{1}{2}\ln(2\pi e \sigma^2)\) nats, increasing with \(\sigma\).
Estimation and inference connect distributions to data. Maximum Likelihood Estimation (MLE) finds \(\hat\theta = \arg\max_\theta \prod_i p(x_i|\theta) = \arg\max_\theta \sum_i \log p(x_i|\theta)\)—log-likelihood is more numerically stable (products cause underflow) but equivalent by monotonicity of log. For a Gaussian: \(\hat\mu = \frac{1}{N}\sum_i x_i\) (sample mean) and \(\hat\sigma^2 = \frac{1}{N}\sum_i(x_i-\hat\mu)^2\) (biased; divide by N-1 for unbiased). Maximum A Posteriori (MAP) incorporates prior: \(\hat\theta = \arg\max[\sum_i \log p(x_i|\theta) + \log p(\theta)]\). A Gaussian prior yields L2 regularization; Laplace prior yields L1—making regularization equivalent to MAP with a chosen prior. Conjugate priors keep the posterior in the same family: Beta-Bernoulli \(\theta \sim \mathrm{Beta}(\alpha,\beta), X|\theta \sim \mathrm{Bern}(\theta) \Rightarrow \theta|X \sim \mathrm{Beta}(\alpha+\mathrm{heads}, \beta+\mathrm{tails})\), Dirichlet-Categorical, Gaussian-Gaussian. Sufficient statistics capture all parameter-relevant information. The moment generating function \(M_X(t) = \mathbb{E}[e^{tX}]\) encodes all moments. The Law of Large Numbers gives \(\bar{X}_N \to \mathbb{E}[X]\); the Central Limit Theorem gives \(\sqrt{N}(\bar{X}_N - \mu) \to \mathcal{N}(0,\sigma^2)\), with \(\bar{X}_N \approx \mathcal{N}(\mu, \sigma^2/N)\) in practice. Monte Carlo estimation approximates \(\mathbb{E}_p[f(x)] \approx \frac{1}{N}\sum_i f(x_i)\) via sampling; importance sampling reweights samples from a tractable proposal \(q\): \(\mathbb{E}_p[f(x)] \approx \frac{1}{N}\sum_i (p(x_i)/q(x_i))f(x_i)\). The bias-variance decomposition \(\mathbb{E}[(\hat f(x) - f(x))^2] = \mathrm{Bias}^2 + \mathrm{Variance} + \sigma^2\) characterizes prediction error, with the classic bias-variance tradeoff (complex models: low bias, high variance; simple: vice versa). The Cramér-Rao lower bound \(\mathrm{Var}(\hat\theta) \geq 1/\mathcal{I}(\theta)\) sets the best achievable precision. Gaussian mixture models \(p(x) = \sum_k \pi_k \mathcal{N}(x|\mu_k, \Sigma_k)\) are fit by the EM algorithm, alternating E-step (posterior \(q^{(t)}(z) = p(z|x,\theta^{(t)})\)) and M-step (\(\theta^{(t+1)} = \arg\max_\theta \mathbb{E}_{q^{(t)}}[\log p(x,z|\theta)]\)). Bootstrap resampling estimates sampling distributions without analytical derivations. The predictive distribution \(p(x_{\mathrm{new}}|x_{1:N}) = \int p(x_{\mathrm{new}}|\theta)p(\theta|x_{1:N})d\theta\) averages predictions over all plausible models, more robust than point estimates.