Probability theory formalizes uncertainty. A sample space \(\Omega\) contains all possible outcomes, and events are subsets of \(\Omega\). For a uniform sample space, the probability of an event equals the number of favorable outcomes divided by the total. The complement rule \(P(A^c) = 1 - P(A)\), the union rule \(P(A \cup B) = P(A) + P(B) - P(A \cap B)\), and conditional probability \(P(A \mid B) = P(A \cap B)/P(B)\) are the basic tools. Independence means \(P(A \cap B) = P(A)P(B)\), and Bayes' theorem \(P(A \mid B) = P(B \mid A)P(A)/P(B)\) updates beliefs about a hypothesis in light of new evidence.
Random variables assign numerical values to outcomes. Discrete variables have probability mass functions (PMFs) listing \(P(X = x)\) for each value, while continuous variables have probability density functions (PDFs) \(f(x)\) where probabilities are areas \(\int_a^b f(x) dx\). The cumulative distribution function \(F(x) = P(X \leq x)\) is the integral of the PDF. The expected value \(E[X] = \sum x P(X=x)\) (or \(\int x f(x) dx\) in the continuous case) is the probability-weighted average, while variance \(\text{Var}(X) = E[(X-\mu)^2]\) measures spread around the mean. Important distributions include Bernoulli (one trial), binomial (\(n\) trials), Poisson (rare events), the normal/Gaussian (ubiquitous from the Central Limit Theorem), and the exponential (waiting times).
Statistics summarizes data and infers properties of populations. The sample mean \(\bar{x}\) and sample variance \(s^2\) (dividing by \(n-1\) for an unbiased estimate) are the foundational descriptive statistics; the median and mode provide alternative measures of center that are robust to outliers. Covariance and correlation measure how two variables move together, with the Pearson correlation \(r\) normalized to \([-1, 1]\). Hypothesis testing evaluates whether data is consistent with a null hypothesis, with the p-value quantifying the probability of observing data this extreme under that null.
In machine learning, these foundations combine to give us the building blocks of models and training. Maximum likelihood estimation finds parameters maximizing \(P(\text{data} \mid \theta)\), while MAP estimation adds a prior \(P(\theta)\) to encode beliefs. Minimizing mean squared error is equivalent to maximum likelihood under Gaussian noise, and minimizing cross-entropy is equivalent to maximum likelihood under a categorical or Bernoulli model. Entropy \(H(X) = -\sum P(x)\log P(x)\) and KL divergence \(D_{KL}(P \| Q) = \sum P(x)\log(P(x)/Q(x))\) measure uncertainty and distributional difference, respectively, with cross-entropy equal to entropy plus KL divergence. Numerical stability matters as much as mathematical correctness: the log-sum-exp trick, log-space likelihood computation, and mixed precision training prevent overflow and underflow in deep networks. The theoretical insights from convexity, gradient descent, regularization, and the bias-variance tradeoff all converge in the practice of designing and training modern learning systems.