Mixture distributions combine several component distributions to model data with heterogeneous structure: an index \(z\) is drawn from Categorical(\(\pi\)), then \(x\) is drawn from \(p(x \mid z)\), giving marginal density \(p(x) = \sum_k \pi_k p_k(x)\). A Gaussian Mixture Model (GMM) is the most common instance, mixing \(K\) multivariate Normals with weights \(\pi_k\), means \(\mu_k\), and covariances \(\Sigma_k\). The component assignment \(z\) is a latent variable — an unobserved quantity that influences observed data — and is learned rather than observed directly. The standard algorithm for fitting GMMs is Expectation-Maximization (EM): the E-step computes posterior responsibilities \(\gamma_{nk} = \pi_k N(x_n; \mu_k, \Sigma_k) / \sum_j \pi_j N(x_n; \mu_j, \Sigma_j)\), and the M-step re-estimates \(\pi_k\), \(\mu_k\), \(\Sigma_k\) from \(\gamma\)-weighted sufficient statistics. EM is a general recipe for maximum likelihood in latent-variable models, and it underlies countless applications from clustering to speech recognition.
Information theory provides the tools to compare distributions and bound quantities that arise in latent-variable modeling. KL divergence \(D_{\text{KL}}(p \| q) = E_{x \sim p}[\log p(x)/q(x)]\) is non-negative, asymmetric, and not a true metric, but it captures how one distribution \(p\) differs from a reference \(q\). Jensen's inequality states that for a convex function \(\varphi\), \(\varphi(E[X]) \le E[\varphi(X)]\); this simple fact drives many variational bounds. Most importantly, the Evidence Lower Bound (ELBO) is defined as \(\text{ELBO} = E_q[\log p(x, z)] + H(q) = \log p(x) - D_{\text{KL}}(q(z) \| p(z \mid x))\); it lower-bounds the log evidence \(\log p(x)\) and is tractable whenever the joint \(p(x, z)\) and a tractable \(q\) can be evaluated, making it the workhorse of variational inference.
Variational inference (VI) approximates an intractable posterior \(p(z \mid x)\) with a tractable \(q_\phi(z)\) by minimizing \(D_{\text{KL}}(q_\phi \| p(\cdot \mid x))\), equivalently maximizing the ELBO. Mean-field VI further restricts \(q\) to a factorized form \(q(z) = \prod_i q_i(z_i)\); this leads to coordinate-ascent updates where each \(q_i\) is set to the log-normalized conditional expectation of \(\log p\) under the current values of the other \(q_j\)'s. When closed-form optimization is impossible, importance sampling offers an alternative: \(E_{p(x)}[f(x)]\) is approximated by \(\frac{1}{N} \sum_i f(x_i) p(x_i)/q(x_i)\) with samples drawn from a proposal \(q\). Variance depends critically on the choice of \(q\), which should ideally have heavier tails than \(|f| p\).