Title: Probability Distributions For Ml — LearnCoachAssist

Q: What is a probability distribution?
A: A function (or rule) that assigns to every outcome in a sample space a non-negative probability, with the total probability over all outcomes summing (or integrating) to 1.

Q: What is a discrete probability distribution?
A: A distribution defined on a countable set of outcomes, specified by a probability mass function (PMF) p(x) with p(x) ≥ 0 and Σ p(x) = 1.

Q: What is a probability mass function (PMF)?
A: A function p(x) that gives the probability of a discrete random variable taking the value x; satisfies p(x) ≥ 0 for all x and Σ_x p(x) = 1.

Q: What is a probability density function (PDF)?
A: A function f(x) for a continuous random variable such that P(a ≤ X ≤ b) = ∫_a^b f(x) dx, with f(x) ≥ 0 and ∫_{-∞}^{∞} f(x) dx = 1.

Q: What is the cumulative distribution function (CDF)?
A: A function F(x) = P(X ≤ x). It is non-decreasing, right-continuous, lim_{x→-∞} F(x) = 0, and lim_{x→∞} F(x) = 1.

Q: What is the relationship between PDF and CDF for a continuous random variable?
A: F(x) = ∫_{-∞}^{x} f(t) dt and, where f is continuous, f(x) = dF/dx.

Q: What is the expected value (mean) of a discrete random variable?
A: E[X] = Σ_x x · p(x), assuming the sum converges; it is a measure of central tendency.

Q: What is the expected value of a continuous random variable?
A: E[X] = ∫_{-∞}^{∞} x · f(x) dx, assuming the integral converges absolutely.

Q: What is the variance of a random variable?
A: Var(X) = E[(X − E[X])^2] = E[X^2] − (E[X])^2; it measures the spread around the mean.

Q: What is the standard deviation?
A: The square root of the variance: σ = √Var(X); it has the same units as X.

Q: What is a Bernoulli distribution?
A: The distribution of a single binary trial: P(X=1) = p, P(X=0) = 1−p. Its mean is p and variance is p(1−p).

Q: What is a Binomial distribution?
A: The number of successes in n independent Bernoulli(p) trials: P(X=k) = C(n,k) p^k (1−p)^{n−k}, with mean np and variance np(1−p).

Q: What is the PMF of the Binomial(n, p) distribution?
A: P(X=k) = (n choose k) p^k (1−p)^{n−k} for k = 0, 1, …, n.

Q: What is a Categorical (Multinoulli) distribution?
A: A generalization of Bernoulli to K outcomes with probabilities p_1, …, p_K summing to 1; samples a single category.

Q: What is a Multinomial distribution?
A: A generalization of Binomial to K categories: counts (X_1,…,X_K) of n independent draws with Σ X_i = n and joint PMF n!/(Π x_i!) Π p_i^{x_i}.

Q: What is a Geometric distribution (number-of-trials form)?
A: The number of trials until the first success in independent Bernoulli(p) trials: P(X=k) = (1−p)^{k−1} p for k = 1, 2, …, with mean 1/p.

Q: What is the "failures-before-success" Geometric distribution?
A: X = number of failures before the first success: P(X=k) = (1−p)^k p for k = 0, 1, 2, …, with mean (1−p)/p.

Q: What is a Negative Binomial distribution?
A: The number of failures before r successes in independent Bernoulli(p) trials: P(X=k) = C(r+k−1, k) p^r (1−p)^k, with mean r(1−p)/p and variance r(1−p)/p^2.

Q: What is a Poisson distribution?
A: The number of events in a fixed interval when events occur independently at a constant rate λ: P(X=k) = e^{−λ} λ^k / k!, with mean λ and variance λ.

Q: When is a Poisson distribution a good model?
A: For rare, independent events at a roughly constant average rate over a fixed interval — e.g. decay events, page hits per minute, or arrival counts.

Q: What is the relationship between Poisson and Binomial as p → 0, n → ∞ with np = λ?
A: P(X=k) ≈ e^{−λ} λ^k / k!; the Binomial approaches the Poisson, justifying Poisson as the "rare events" limit.

Q: What is a Uniform distribution on [a, b]?
A: Continuous distribution with constant density f(x) = 1/(b−a) for x ∈ [a,b], else 0; mean (a+b)/2 and variance (b−a)^2/12.

Q: What is a Bernoulli + Uniform trick to convert random bits to a uniform sample?
A: To sample X uniformly on {0,…,2^k−1}, draw k independent Bernoulli(0.5) and interpret the resulting bit string as a binary integer.

Q: What is a Normal (Gaussian) distribution?
A: A continuous distribution with PDF f(x) = (1/(σ√(2π))) exp(−(x−μ)^2/(2σ^2)), parameterized by mean μ and variance σ^2.

Q: What is the standard normal distribution?
A: A Normal with μ = 0 and σ^2 = 1; its CDF is denoted Φ(z) and PDF φ(z) = (1/√(2π)) e^{−z^2/2}.

Q: What is the 68-95-99.7 rule for the Normal?
A: Approximately 68% of mass within 1σ, 95% within 2σ, and 99.7% within 3σ of the mean.

Q: What is the Central Limit Theorem (CLT)?
A: The (properly normalized) sum of n i.i.d. random variables with finite mean μ and variance σ^2 converges in distribution to Normal(μ, σ^2/n) as n → ∞.

Q: Why is the CLT crucial for machine learning?
A: It justifies treating averages, losses, and estimator errors as approximately Normal, enabling confidence intervals, z-tests, and Gaussian modeling of aggregate behavior.

Q: What is the Box-Muller transform?
A: A method to generate two independent standard normal samples from two independent U(0,1) uniforms: Z0 = √(−2 ln U1) cos(2π U2), Z1 = √(−2 ln U1) sin(2π U2).

Q: What is a Log-Normal distribution?
A: A positive random variable X whose log is Normal(μ, σ^2); its density is f(x) = (1/(xσ√(2π))) exp(−(ln x − μ)^2/(2σ^2)) for x > 0; mean e^{μ+σ^2/2}.

Q: What is an Exponential distribution?
A: The time between events in a Poisson process of rate λ: PDF f(t) = λ e^{−λt} for t ≥ 0; mean 1/λ, variance 1/λ^2, and it is memoryless.

Q: What does memoryless mean for the Exponential distribution?
A: P(X > s + t | X > s) = P(X > t) for s, t ≥ 0; equivalently, the conditional distribution of remaining time does not depend on elapsed time.

Q: What is the Gamma distribution?
A: A two-parameter family with shape k > 0 and rate θ > 0 (or scale β = 1/θ): PDF f(x) = x^{k−1} e^{−x/θ} / (Γ(k) θ^k) for x > 0; mean kθ, variance kθ^2.

Q: What is the relationship between Exponential and Gamma?
A: Gamma(k=1, θ) is the Exponential(1/θ) distribution; Gamma models the sum of k i.i.d. Exponential(1/θ) waiting times.

Q: What is the Beta distribution?
A: A continuous distribution on (0,1) with PDF f(p) = p^{α−1} (1−p)^{β−1} / B(α,β), where B is the Beta function; mean α/(α+β).

Q: Why is the Beta distribution widely used in ML?
A: It is the conjugate prior for the Bernoulli, Binomial, Geometric, and (with α,β > 1) provides a flexible distribution on probabilities.

Q: What is the Dirichlet distribution?
A: A multivariate generalization of Beta on the K-simplex: f(π) = (1/B(α)) Π π_i^{α_i−1} for π_i ≥ 0, Σ π_i = 1; it is the conjugate prior of the Multinomial.

Q: What is the Dirichlet-multinomial conjugacy?
A: Multinomial likelihood with Dirichlet(α) prior yields a Dirichlet(α + counts) posterior over category probabilities; this is foundational for Bayesian text models like LDA.

Q: What is a mixture distribution?
A: A distribution formed by drawing a component index z ~ Categorical(π) and then drawing x | z ~ p(x | z); the marginal density is p(x) = Σ_k π_k p_k(x).

Q: What is a Gaussian Mixture Model (GMM)?
A: A mixture of K multivariate Normal components with weights π_k, means μ_k, covariances Σ_k: p(x) = Σ_k π_k N(x; μ_k, Σ_k).

Q: What is the Expectation-Maximization (EM) algorithm for GMMs?
A: An iterative algorithm: E-step computes posterior responsibilities γ_{nk} = π_k N(x_n; μ_k, Σ_k) / Σ_j π_j N(x_n; μ_j, Σ_j); M-step re-estimates π_k, μ_k, Σ_k from γ-weighted sufficient statistics.

Q: What is a latent variable?
A: An unobserved random variable that influences observed data; in GMMs, the component assignment z is latent and learned via EM.

Q: What is KL divergence and what does it measure?
A: D_KL(p || q) = E_{x∼p}[log p(x)/q(x)] ≥ 0; it measures how one distribution p differs from a reference q, but is asymmetric and not a true metric.

Q: What is Jensen's inequality?
A: For a convex function φ, φ(E[X]) ≤ E[φ(X)]; used to derive the ELBO and variational inference bounds.

Q: What is the Evidence Lower Bound (ELBO)?
A: ELBO = E_q[log p(x,z)] + H(q) = log p(x) − D_KL(q(z) || p(z|x)); it is a tractable lower bound on the log evidence used in variational inference.

Q: What is the difference between discriminative and generative models in distribution terms?
A: Discriminative models p(y|x) directly model the conditional; generative models model the joint p(x, y) or p(x), from which p(y|x) can be derived via Bayes' rule.

Q: What is the softmax function and where does it arise?
A: softmax(z_i) = e^{z_i} / Σ_j e^{z_j} maps real-valued scores to a Categorical distribution; it is the multinomial logit / normalized exponential and arises naturally from the multinomial logistic regression likelihood.

Q: What is the sigmoid (logistic) function?
A: σ(z) = 1/(1 + e^{−z}); it maps a real number to (0,1) and parameterizes Bernoulli probabilities in logistic regression.

Q: What is the logistic (sigmoid) distribution?
A: A continuous distribution with PDF f(x) = e^{−(x−μ)/s} / (s(1 + e^{−(x−μ)/s})^2), mean μ, variance s^2 π^2/3; its CDF is the logistic (sigmoid) function.

Q: What is the Laplace (double-exponential) distribution?
A: f(x | μ, b) = (1/(2b)) exp(−|x−μ|/b); mean μ, variance 2b^2; heavier tails than Normal and often used for sparse / robust regression (L1 prior on coefficients).

Q: What is the Student's t-distribution?
A: A continuous distribution with ν degrees of freedom, mean 0 (for ν > 1) and variance ν/(ν−2) for ν > 2; it has heavier tails than the Normal and approaches Normal as ν → ∞.

Q: What is the Cauchy (Lorentz) distribution?
A: A t-distribution with ν = 1 degree of freedom; PDF f(x) = 1/(π γ (1 + ((x−x0)/γ)^2)). It has undefined mean and variance and is a classic example of a heavy-tailed distribution.

Q: What is a heavy-tailed distribution?
A: A distribution whose tail decays slower than exponential (e.g. power-law P(X>x) ~ x^{−α}); it can produce extreme outliers and may have infinite variance.

Q: What is a power-law distribution?
A: A distribution whose PDF or PMF behaves as p(x) ∝ x^{−α} for large x; common in degrees of social networks, word frequencies (Zipf), and city sizes.

Q: What is a Pareto distribution?
A: A power-law with PDF f(x) = α x_m^α / x^{α+1} for x ≥ x_m; mean α x_m/(α−1) for α>1, variance finite only for α>2; widely used to model wealth and city size (the "80/20" rule).

Q: What is the Zipf distribution?
A: A discrete power-law where the k-th most frequent item has probability ∝ 1/k^s; the basis of Zipf's law in word frequencies.

Q: What is the chi-squared (χ²) distribution?
A: The distribution of Z_1^2 + … + Z_k^2 where Z_i are i.i.d. standard Normal; it is Gamma(k/2, 2) and used in goodness-of-fit and likelihood-ratio tests.

Q: What is the F-distribution?
A: The ratio (χ²_d1 / d1) / (χ²_d2 / d2) of two independent scaled chi-squared variates; used in ANOVA and in comparing variances.

Q: What is the Weibull distribution?
A: A generalization of Exponential with shape k > 0 and scale λ: F(t) = 1 − exp(−(t/λ)^k) for t ≥ 0; used in reliability/survival analysis and for modeling time-to-failure.

Q: What is a survival function?
A: S(t) = P(T > t) = 1 − F(t); it gives the probability of "surviving" past time t and is central to survival analysis.

Q: What is the hazard function?
A: h(t) = f(t)/S(t), the instantaneous failure rate at time t conditional on survival to t; fully characterizes the distribution of a non-negative random variable together with S(0).

Q: What is the Weibull hazard shape?
A: For shape k < 1 the hazard decreases (infant mortality), for k = 1 it is constant (Exponential), and for k > 1 it is increasing (wear-out).

Q: What is the Rayleigh distribution?
A: A Weibull with shape 2, scale σ: f(x) = (x/σ^2) exp(−x^2/(2σ^2)) for x ≥ 0; used to model magnitudes of 2D Gaussian vectors, e.g. signal envelopes.

Q: What is the Maxwell-Boltzmann distribution?
A: A chi distribution with 3 degrees of freedom; it describes the distribution of speeds of particles in an ideal gas.

Q: What is a multivariate Gaussian (Normal) distribution?
A: X ∈ R^d with density f(x) = (2π)^{−d/2} |Σ|^{−1/2} exp(−½ (x−μ)^T Σ^{−1} (x−μ)); mean μ, covariance Σ (must be positive definite).

Q: What does the Mahalanobis distance measure?
A: d(x, μ) = √((x−μ)^T Σ^{−1} (x−μ)); the distance in units of standard deviations after whitening by Σ.

Q: What is a singular Gaussian (degenerate Normal)?
A: A multivariate Normal whose covariance Σ is singular (non-invertible); its mass lies on a lower-dimensional affine subspace and the PDF is undefined in the ambient space.

Q: What is the "reparameterization trick" for Gaussians?
A: Writing x = μ + σ ⊙ ε with ε ~ N(0, I) so that gradients of E_{x~N(μ,σ²)}[f(x)] can flow through μ and σ — used in variational autoencoders.

Q: What is the Wishart distribution?
A: The distribution of sums (and more generally, sample covariances) of outer products of multivariate Gaussian vectors; it generalizes chi-squared to matrices and is the conjugate prior for the inverse covariance of a multivariate Normal.

Q: What is the inverse-Wishart distribution?
A: The conjugate prior for a covariance matrix Σ; if Σ ~ Inv-Wishart(Ψ, ν) and data are i.i.d. multivariate Normal with known mean, the posterior is also inverse-Wishart.

Q: What is the LKJ prior for correlation matrices?
A: A prior on correlation matrices indexed by a shape η ≥ 0; η = 1 is uniform on the space of correlation matrices and larger η concentrates density near the identity.

Q: What is an isotropic Gaussian?
A: A multivariate Normal with Σ = σ^2 I_d; its density depends only on Euclidean distance and is rotationally symmetric.

Q: What is whitening?
A: A linear transformation z = L^{−1}(x − μ) that maps a centered multivariate Normal with covariance Σ = L L^T to a standard Normal; typically L is the Cholesky factor of Σ.

Q: What is a Bernoulli Naive Bayes classifier in distributional terms?
A: A generative model assuming features are conditionally independent given class y ~ Categorical: P(x_i | y = c) = Bernoulli(p_{c,i}); the joint is the product of independent Bernoullis.

Q: What is a Gaussian Naive Bayes classifier?
A: A generative model assuming features are conditionally independent given class y, with P(x_i | y = c) = N(x_i; μ_{c,i}, σ²_{c,i}); decision rule is the maximum posterior class.

Q: What is a Gaussian Discriminant Analysis (GDA)?
A: A generative model with class-conditional multivariate Gaussians p(x|y=k) = N(x; μ_k, Σ_k) and class prior p(y=k) = π_k; the Bayes-optimal decision boundary is quadratic (linear when Σ_k are equal).

Q: What is a Linear Discriminant Analysis (LDA) classifier?
A: GDA with the additional assumption that all classes share a common covariance Σ; the resulting Bayes decision boundary is linear in x.

Q: What is Quadratic Discriminant Analysis (QDA)?
A: GDA without the equal-covariance assumption; the Bayes decision boundary is a quadratic function of x.

Q: What is the connection between LDA and Fisher's linear discriminant?
A: Both find a low-dimensional projection that maximizes class separability; Fisher's criterion gives a closed-form projection, while LDA arises from the generative model with shared covariance.

Q: What is a Multinomial Logistic Regression model?
A: Discriminative model for K classes: p(y=k | x) = exp(w_k^T x + b_k) / Σ_j exp(w_j^T x + b_j); equivalent to a Categorical likelihood with softmax parameterization.

Q: What is the categorical cross-entropy loss?
A: Loss = −Σ_n Σ_k y_{n,k} log p(y_{n,k} | x_n); it is the negative log-likelihood of the Categorical distribution and is the standard loss for multinomial logistic regression / softmax classifiers.

Q: What is the binary cross-entropy loss?
A: Loss = −(1/N) Σ_n [y_n log p(x_n) + (1−y_n) log(1−p(x_n))]; it is the negative log-likelihood under a Bernoulli likelihood with p = σ(w^T x + b).

Q: What is the MSE loss in distributional terms?
A: MSE = (1/N) Σ (y_n − f(x_n))^2; it is the negative log-likelihood of a Gaussian with fixed variance, up to a constant.

Q: What is the MLE principle?
A: Maximum likelihood estimation chooses parameters θ to maximize the log-likelihood L(θ) = Σ_n log p(x_n | θ); equivalently it minimizes KL between the empirical distribution and p(· | θ) up to a constant.

Q: What is the MAP principle?
A: Maximum a posteriori estimation maximizes log p(θ | D) ∝ log p(D | θ) + log p(θ); it adds a prior regularization term to the MLE objective.

Q: What is conjugate prior?
A: A prior p(θ) such that, when combined with a likelihood from a given family, the posterior p(θ | D) is in the same family as the prior (e.g. Beta–Bernoulli, Dirichlet–Multinomial, Normal–Normal, Gamma–Poisson, Inv-Wishart–Normal).

Q: What is the Jeffreys prior?
A: A non-informative prior proportional to √|I(θ)|, where I(θ) is the Fisher information; it is invariant to reparameterization.

Q: What is the Boltzmann (Gibbs) distribution?
A: p(x) = exp(−E(x)/T) / Z with partition function Z = Σ_x exp(−E(x)/T); it is the canonical distribution in statistical physics and underlies energy-based models.

Q: What is an energy-based model (EBM)?
A: A model of the form p_θ(x) = exp(−E_θ(x)) / Z(θ) where E_θ is a learned energy function and Z(θ) = ∫ exp(−E_θ(x)) dx is a (typically intractable) normalizer.

Q: What is a restricted Boltzmann machine (RBM)?
A: A two-layer undirected graphical model with visible units v and binary hidden units h; joint p(v, h) ∝ exp(b^T v + c^T h + v^T W h); trained by contrastive divergence.

Q: What is the connection between softmax and the Boltzmann distribution?
A: Softmax(z) = exp(z_i)/Σ exp(z_j) is a Gibbs distribution over discrete states with "energy" −z_i and temperature 1; sampling from a softmax classifier with temperature T uses exp(z_i/T).

Q: What is a Markov chain?
A: A stochastic process where P(X_t | X_{1:t−1}) = P(X_t | X_{t−1}); the future is conditionally independent of the past given the present.

Q: What is a stationary distribution of a Markov chain?
A: A distribution π such that π = πP; under irreducibility and aperiodicity, π exists, is unique, and is the long-run fraction of time the chain spends in each state.

Q: What is the detailed balance condition?
A: π_i P_{ij} = π_j P_{ji} for all i, j; sufficient (with irreducibility) to guarantee that π is the stationary distribution.

Q: What is the Metropolis-Hastings algorithm?
A: An MCMC method: propose y from q(y|x), accept with probability min(1, π(y) q(x|y) / (π(x) q(y|x))); the resulting chain has stationary distribution π.

Q: What is Gibbs sampling?
A: An MCMC method that samples each variable from its full conditional given the others; a special case of Metropolis-Hastings with proposal equal to the conditional and acceptance 1.

Q: What is Hamiltonian Monte Carlo (HMC)?
A: An MCMC method that uses gradient information to propose distant moves that are accepted with high probability; effective for high-dimensional, smooth targets.

Q: What is a score function in a probabilistic model?
A: ∇_θ log p_θ(x); it is the gradient of the log-likelihood with respect to parameters and is the basis of REINFORCE and likelihood-ratio gradient estimators.

Q: What is a reparameterized gradient estimator?
A: An estimator that writes x = g_θ(ε) with ε ~ q(ε) noise-free, so ∇_θ E_q[f(x)] = E_ε[∇_θ f(g_θ(ε))]; it has lower variance than score-function estimators in many settings.

Q: What is the REINFORCE estimator?
A: A score-function gradient estimator: ∇_θ E_{p_θ}[f(x)] = E_{p_θ}[f(x) ∇_θ log p_θ(x)]; unbiased but typically high-variance, reducible via baselines / control variates.

Q: What is a baseline (control variate) for variance reduction?
A: A function b(x) subtracted from f(x) in the REINFORCE estimator; valid because E[∇_θ log p_θ(x) · b(x)] = 0 when b does not depend on θ or sampling.

Q: What is a variational autoencoder (VAE)?
A: A latent-variable model p_θ(x | z) with z ~ N(0, I) prior, trained by maximizing the ELBO using an inference network q_φ(z | x); gradients obtained via the reparameterization trick.

Q: What is a normalizing flow?
A: A sequence of invertible, differentiable transformations f = f_K ∘ … ∘ f_1 mapping a simple base density (e.g. N(0, I)) to a complex one; log p(x) = log p_z(f^{-1}(x)) + Σ log |det(∂f_i^{−1}/∂x)|.

Q: What is a diffusion model in distributional terms?
A: A model that learns to reverse a fixed noising Markov chain that gradually adds Gaussian noise to data; training minimizes a denoising score-matching loss on p(x_t | x_0) ~ N(√α̅_t x_0, (1−α̅_t) I).

Q: What is a score-matching objective?
A: An objective that matches ∇_x log p_θ(x) to the data score ∇_x log p_data(x) by minimizing E_{p_data}[ ½ ||∇_x log p_θ(x) − ∇_x log p_data(x)||^2 ]; the constant involving the Hessian of p_data cancels.

Q: What is denoising score matching?
A: A tractable surrogate for score matching that corrupts data with Gaussian noise and matches the score of the noise-conditional model; basis of modern diffusion model training.

Q: What is a latent Dirichlet allocation (LDA) topic model?
A: A generative model where each document has a topic-mixing vector θ_d ~ Dirichlet(α), each topic k has a word distribution φ_k ~ Dirichlet(β), and each word is drawn by choosing z ~ Categorical(θ_d) and w ~ Categorical(φ_z).

Q: What is Bayesian linear regression?
A: A linear model y = w^T x + ε with ε ~ N(0, σ^2) and Gaussian prior w ~ N(0, τ^2 I); posterior p(w | D) is Gaussian with closed-form mean (X^T X + (σ^2/τ^2) I)^{−1} X^T y.

Q: What is the kernel density estimator (KDE)?
A: An estimator f̂(x) = (1/(n h)) Σ_i K((x − x_i)/h) that places a smooth kernel K (bandwidth h) on each data point; an empirical approximation of the unknown density.

Q: What is the bias-variance tradeoff in density estimation?
A: Smoother models (large bandwidth h, simple parametric families) yield higher bias but lower variance, while complex models (small h, flexible families) yield lower bias but higher variance and risk of overfitting.

Q: What is the AIC / BIC model selection criterion?
A: AIC = 2k − 2 log L̂; BIC = k log n − 2 log L̂; they penalize model complexity (k parameters) and BIC additionally penalizes more strongly with sample size n.

Q: What is the difference between MLE and MAP for a Gaussian mean with known variance?
A: MLE returns x̄; MAP under a Gaussian prior N(μ_0, τ^2) returns (n τ^2 x̄ + σ^2 μ_0) / (n τ^2 + σ^2), a precision-weighted shrinkage estimate.

Q: What is the difference between frequentist and Bayesian inference for distributions?
A: Frequentists treat the true parameter as fixed and use sampling distributions; Bayesians treat parameters as random with a prior and update a posterior via Bayes' rule, yielding full predictive distributions.

Q: What is the posterior predictive distribution?
A: p(x_new | D) = ∫ p(x_new | θ) p(θ | D) dθ; the Bayesian average of likelihoods over the posterior, used for prediction under uncertainty.

Q: What is the prior predictive distribution?
A: p(x) = ∫ p(x | θ) p(θ) dθ; the marginal distribution of data induced by the prior on parameters.

Q: What is the conjugacy advantage in posterior computation?
A: Conjugate priors yield closed-form posteriors; for non-conjugate models, one resorts to MCMC, variational inference, or Laplace / EP approximations.

Q: What is variational inference?
A: A method that approximates an intractable posterior p(z|x) with a tractable q_φ(z) by minimizing D_KL(q_φ || p(·|x)), equivalently maximizing the ELBO.

Q: What is mean-field variational inference?
A: VI that restricts q to a fully factorized family q(z) = Π_i q_i(z_i); leads to coordinate-ascent updates where each q_i is set to the log-normalized conditional expectation of log p under the others.

Q: What is importance sampling?
A: An estimator E_{p(x)}[f(x)] ≈ (1/N) Σ_i f(x_i) p(x_i)/q(x_i) with x_i ~ q; the choice of proposal q is critical for variance — it should have heavier tails than |f| p.

Q: What is a reweighted wake-sleep algorithm?
A: A training method for latent-variable models that alternates: wake phase fits an inference network to samples from the model; sleep phase fits the generative model to samples from the inference network.