Parameter estimation can be framed either in frequentist or Bayesian terms. Maximum likelihood estimation (MLE) chooses parameters \(\theta\) to maximize the log-likelihood \(L(\theta) = \sum_n \log p(x_n \mid \theta)\), equivalently minimizing the KL divergence between the empirical distribution and \(p(\cdot \mid \theta)\) up to a constant. Maximum a posteriori (MAP) estimation adds a prior term: it maximizes \(\log p(\theta \mid D) \propto \log p(D \mid \theta) + \log p(\theta)\). For instance, when estimating a Gaussian mean with known variance \(\sigma^2\) under a Gaussian prior \(N(\mu_0, \tau^2)\), MLE returns the sample mean \(\bar{x}\), while MAP returns the precision-weighted shrinkage estimator \((n \tau^2 \bar{x} + \sigma^2 \mu_0) / (n \tau^2 + \sigma^2)\), interpolating between data and prior. Frequentists treat the true parameter as fixed and reason about sampling distributions, whereas Bayesians treat parameters as random variables, place priors over them, and update to posteriors via Bayes' rule, yielding full predictive distributions.
Conjugate priors are a cornerstone of Bayesian computation: a prior \(p(\theta)\) is conjugate to a likelihood if the posterior \(p(\theta \mid D)\) lies in the same family as the prior. Standard pairings include Beta–Bernoulli, Dirichlet–Multinomial, Gamma–Poisson, and Normal–Normal, as well as inverse-Wishart for multivariate Normal covariances. For example, with a Bernoulli likelihood and Beta(\(\alpha, \beta\)) prior, the posterior is Beta(\(\alpha +\) successes, \(\beta +\) failures); analogously, a Multinomial likelihood combined with a Dirichlet(\(\alpha\)) prior yields a Dirichlet(\(\alpha +\) counts) posterior, a relationship that underpins Bayesian text models like Latent Dirichlet Allocation. When conjugacy fails, posterior computation requires approximate methods: MCMC, variational inference, or Laplace / expectation-propagation approximations.
The choice of prior encodes prior beliefs and affects posterior behavior. The Jeffreys prior is non-informative in a precise sense: it is proportional to \(\sqrt{|I(\theta)|}\), where \(I(\theta)\) is the Fisher information, and is invariant under reparameterization. The posterior predictive distribution \(p(x_{\text{new}} \mid D) = \int p(x_{\text{new}} \mid \theta) p(\theta \mid D)\, d\theta\) averages likelihoods over the posterior to make predictions under uncertainty, while the prior predictive \(p(x) = \int p(x \mid \theta) p(\theta)\, d\theta\) gives the marginal distribution induced by the prior alone. Bayesian linear regression illustrates these ideas concretely: under \(y = w^T x + \varepsilon\) with \(\varepsilon \sim N(0, \sigma^2)\) and prior \(w \sim N(0, \tau^2 I)\), the posterior over weights is itself Gaussian with closed-form mean \((X^T X + (\sigma^2/\tau^2) I)^{-1} X^T y\), and the precision term \(\sigma^2/\tau^2\) acts as an L2 regularizer.