Probability Distributions for ML Practice Exam

Question 1 Probability Distributions for ML

What is the difference between MLE and MAP for a Gaussian mean with known variance?

A sequence of invertible, differentiable transformations f = f_K ∘ … ∘ f_1 mapping a simple base density (e.g. N(0, I)) to a complex one; log p(x) = log p_z(f^{-1}(x)) + Σ log |det(∂f_i^{−1}/∂x)|. An MCMC method that uses gradient information to propose distant moves that are accepted with high probability; effective for high-dimensional, smooth targets. Var(X) = E[(X − E[X])^2] = E[X^2] − (E[X])^2; it measures the spread around the mean. MLE returns x̄; MAP under a Gaussian prior N(μ_0, τ^2) returns (n τ^2 x̄ + σ^2 μ_0) / (n τ^2 + σ^2), a precision-weighted shrinkage estimate.

Question 2 Probability Distributions for ML

What is the Expectation-Maximization (EM) algorithm for GMMs?

An iterative algorithm: E-step computes posterior responsibilities γ_{nk} = π_k N(x_n; μ_k, Σ_k) / Σ_j π_j N(x_n; μ_j, Σ_j); M-step re-estimates π_k, μ_k, Σ_k from γ-weighted sufficient statistics. A model that learns to reverse a fixed noising Markov chain that gradually adds Gaussian noise to data; training minimizes a denoising score-matching loss on p(x_t | x_0) ~ N(√α̅_t x_0, (1−α̅_t) I). MSE = (1/N) Σ (y_n − f(x_n))^2; it is the negative log-likelihood of a Gaussian with fixed variance, up to a constant. A generalization of Bernoulli to K outcomes with probabilities p_1, …, p_K summing to 1; samples a single category.

Question 3 Probability Distributions for ML

What is a mixture distribution?

A continuous distribution with PDF f(x) = (1/(σ√(2π))) exp(−(x−μ)^2/(2σ^2)), parameterized by mean μ and variance σ^2. Maximum a posteriori estimation maximizes log p(θ | D) ∝ log p(D | θ) + log p(θ); it adds a prior regularization term to the MLE objective. Discriminative model for K classes: p(y=k | x) = exp(w_k^T x + b_k) / Σ_j exp(w_j^T x + b_j); equivalent to a Categorical likelihood with softmax parameterization. A distribution formed by drawing a component index z ~ Categorical(π) and then drawing x | z ~ p(x | z); the marginal density is p(x) = Σ_k π_k p_k(x).

Question 4 Probability Distributions for ML

What is the inverse-Wishart distribution?

The conjugate prior for a covariance matrix Σ; if Σ ~ Inv-Wishart(Ψ, ν) and data are i.i.d. multivariate Normal with known mean, the posterior is also inverse-Wishart. A generative model where each document has a topic-mixing vector θ_d ~ Dirichlet(α), each topic k has a word distribution φ_k ~ Dirichlet(β), and each word is drawn by choosing z ~ Categorical(θ_d) and w ~ Categorical(φ_z). Loss = −(1/N) Σ_n [y_n log p(x_n) + (1−y_n) log(1−p(x_n))]; it is the negative log-likelihood under a Bernoulli likelihood with p = σ(w^T x + b). P(X > s + t | X > s) = P(X > t) for s, t ≥ 0; equivalently, the conditional distribution of remaining time does not depend on elapsed time.

Question 5 Probability Distributions for ML

What is a Categorical (Multinoulli) distribution?

The number of trials until the first success in independent Bernoulli(p) trials: P(X=k) = (1−p)^{k−1} p for k = 1, 2, …, with mean 1/p. A generalization of Bernoulli to K outcomes with probabilities p_1, …, p_K summing to 1; samples a single category. The distribution of a single binary trial: P(X=1) = p, P(X=0) = 1−p. Its mean is p and variance is p(1−p). A sequence of invertible, differentiable transformations f = f_K ∘ … ∘ f_1 mapping a simple base density (e.g. N(0, I)) to a complex one; log p(x) = log p_z(f^{-1}(x)) + Σ log |det(∂f_i^{−1}/∂x)|.

Question 6 Probability Distributions for ML

What is the F-distribution?

F(x) = ∫_{-∞}^{x} f(t) dt and, where f is continuous, f(x) = dF/dx. The ratio (χ²_d1 / d1) / (χ²_d2 / d2) of two independent scaled chi-squared variates; used in ANOVA and in comparing variances. Softmax(z) = exp(z_i)/Σ exp(z_j) is a Gibbs distribution over discrete states with "energy" −z_i and temperature 1; sampling from a softmax classifier with temperature T uses exp(z_i/T). MSE = (1/N) Σ (y_n − f(x_n))^2; it is the negative log-likelihood of a Gaussian with fixed variance, up to a constant.

Question 7 Probability Distributions for ML

What is the connection between LDA and Fisher's linear discriminant?

An objective that matches ∇_x log p_θ(x) to the data score ∇_x log p_data(x) by minimizing E_{p_data}[ ½ ||∇_x log p_θ(x) − ∇_x log p_data(x)||^2 ]; the constant involving the Hessian of p_data cancels. Gamma(k=1, θ) is the Exponential(1/θ) distribution; Gamma models the sum of k i.i.d. Exponential(1/θ) waiting times. P(X > s + t | X > s) = P(X > t) for s, t ≥ 0; equivalently, the conditional distribution of remaining time does not depend on elapsed time. Both find a low-dimensional projection that maximizes class separability; Fisher's criterion gives a closed-form projection, while LDA arises from the generative model with shared covariance.

Question 8 Probability Distributions for ML

What is the Gamma distribution?

A generative model with class-conditional multivariate Gaussians p(x|y=k) = N(x; μ_k, Σ_k) and class prior p(y=k) = π_k; the Bayes-optimal decision boundary is quadratic (linear when Σ_k are equal). A continuous distribution on (0,1) with PDF f(p) = p^{α−1} (1−p)^{β−1} / B(α,β), where B is the Beta function; mean α/(α+β). A stochastic process where P(X_t | X_{1:t−1}) = P(X_t | X_{t−1}); the future is conditionally independent of the past given the present. A two-parameter family with shape k > 0 and rate θ > 0 (or scale β = 1/θ): PDF f(x) = x^{k−1} e^{−x/θ} / (Γ(k) θ^k) for x > 0; mean kθ, variance kθ^2.

Question 9 Probability Distributions for ML

What is the "reparameterization trick" for Gaussians?

A positive random variable X whose log is Normal(μ, σ^2); its density is f(x) = (1/(xσ√(2π))) exp(−(ln x − μ)^2/(2σ^2)) for x > 0; mean e^{μ+σ^2/2}. Writing x = μ + σ ⊙ ε with ε ~ N(0, I) so that gradients of E_{x~N(μ,σ²)}[f(x)] can flow through μ and σ — used in variational autoencoders. A function (or rule) that assigns to every outcome in a sample space a non-negative probability, with the total probability over all outcomes summing (or integrating) to 1. A two-parameter family with shape k > 0 and rate θ > 0 (or scale β = 1/θ): PDF f(x) = x^{k−1} e^{−x/θ} / (Γ(k) θ^k) for x > 0; mean kθ, variance kθ^2.

Question 10 Probability Distributions for ML

What is the Zipf distribution?

A function p(x) that gives the probability of a discrete random variable taking the value x; satisfies p(x) ≥ 0 for all x and Σ_x p(x) = 1. Gamma(k=1, θ) is the Exponential(1/θ) distribution; Gamma models the sum of k i.i.d. Exponential(1/θ) waiting times. A discrete power-law where the k-th most frequent item has probability ∝ 1/k^s; the basis of Zipf's law in word frequencies. π_i P_{ij} = π_j P_{ji} for all i, j; sufficient (with irreducibility) to guarantee that π is the stationary distribution.

Question 11 Probability Distributions for ML

What is a probability mass function (PMF)?

An MCMC method: propose y from q(y|x), accept with probability min(1, π(y) q(x|y) / (π(x) q(y|x))); the resulting chain has stationary distribution π. The time between events in a Poisson process of rate λ: PDF f(t) = λ e^{−λt} for t ≥ 0; mean 1/λ, variance 1/λ^2, and it is memoryless. A function p(x) that gives the probability of a discrete random variable taking the value x; satisfies p(x) ≥ 0 for all x and Σ_x p(x) = 1. A generative model where each document has a topic-mixing vector θ_d ~ Dirichlet(α), each topic k has a word distribution φ_k ~ Dirichlet(β), and each word is drawn by choosing z ~ Categorical(θ_d) and w ~ Categorical(φ_z).

Question 12 Probability Distributions for ML

What is a reweighted wake-sleep algorithm?

A non-informative prior proportional to √|I(θ)|, where I(θ) is the Fisher information; it is invariant to reparameterization. A training method for latent-variable models that alternates: wake phase fits an inference network to samples from the model; sleep phase fits the generative model to samples from the inference network. Approximately 68% of mass within 1σ, 95% within 2σ, and 99.7% within 3σ of the mean. The time between events in a Poisson process of rate λ: PDF f(t) = λ e^{−λt} for t ≥ 0; mean 1/λ, variance 1/λ^2, and it is memoryless.

Question 13 Probability Distributions for ML

What is a Poisson distribution?

f(x | μ, b) = (1/(2b)) exp(−|x−μ|/b); mean μ, variance 2b^2; heavier tails than Normal and often used for sparse / robust regression (L1 prior on coefficients). Loss = −Σ_n Σ_k y_{n,k} log p(y_{n,k} | x_n); it is the negative log-likelihood of the Categorical distribution and is the standard loss for multinomial logistic regression / softmax classifiers. P(X > s + t | X > s) = P(X > t) for s, t ≥ 0; equivalently, the conditional distribution of remaining time does not depend on elapsed time. The number of events in a fixed interval when events occur independently at a constant rate λ: P(X=k) = e^{−λ} λ^k / k!, with mean λ and variance λ.

Question 14 Probability Distributions for ML

What is a Gaussian Discriminant Analysis (GDA)?

A model that learns to reverse a fixed noising Markov chain that gradually adds Gaussian noise to data; training minimizes a denoising score-matching loss on p(x_t | x_0) ~ N(√α̅_t x_0, (1−α̅_t) I). The time between events in a Poisson process of rate λ: PDF f(t) = λ e^{−λt} for t ≥ 0; mean 1/λ, variance 1/λ^2, and it is memoryless. P(X=k) = (n choose k) p^k (1−p)^{n−k} for k = 0, 1, …, n. A generative model with class-conditional multivariate Gaussians p(x|y=k) = N(x; μ_k, Σ_k) and class prior p(y=k) = π_k; the Bayes-optimal decision boundary is quadratic (linear when Σ_k are equal).

Question 15 Probability Distributions for ML

What is the chi-squared (χ²) distribution?

A two-layer undirected graphical model with visible units v and binary hidden units h; joint p(v, h) ∝ exp(b^T v + c^T h + v^T W h); trained by contrastive divergence. An iterative algorithm: E-step computes posterior responsibilities γ_{nk} = π_k N(x_n; μ_k, Σ_k) / Σ_j π_j N(x_n; μ_j, Σ_j); M-step re-estimates π_k, μ_k, Σ_k from γ-weighted sufficient statistics. A continuous distribution with ν degrees of freedom, mean 0 (for ν > 1) and variance ν/(ν−2) for ν > 2; it has heavier tails than the Normal and approaches Normal as ν → ∞. The distribution of Z_1^2 + … + Z_k^2 where Z_i are i.i.d. standard Normal; it is Gamma(k/2, 2) and used in goodness-of-fit and likelihood-ratio tests.

Question 16 Probability Distributions for ML

What is a Binomial distribution?

The number of successes in n independent Bernoulli(p) trials: P(X=k) = C(n,k) p^k (1−p)^{n−k}, with mean np and variance np(1−p). A multivariate Normal with Σ = σ^2 I_d; its density depends only on Euclidean distance and is rotationally symmetric. Loss = −(1/N) Σ_n [y_n log p(x_n) + (1−y_n) log(1−p(x_n))]; it is the negative log-likelihood under a Bernoulli likelihood with p = σ(w^T x + b). MLE returns x̄; MAP under a Gaussian prior N(μ_0, τ^2) returns (n τ^2 x̄ + σ^2 μ_0) / (n τ^2 + σ^2), a precision-weighted shrinkage estimate.

Question 17 Probability Distributions for ML

What does memoryless mean for the Exponential distribution?

Smoother models (large bandwidth h, simple parametric families) yield higher bias but lower variance, while complex models (small h, flexible families) yield lower bias but higher variance and risk of overfitting. Multinomial likelihood with Dirichlet(α) prior yields a Dirichlet(α + counts) posterior over category probabilities; this is foundational for Bayesian text models like LDA. P(X > s + t | X > s) = P(X > t) for s, t ≥ 0; equivalently, the conditional distribution of remaining time does not depend on elapsed time. Maximum a posteriori estimation maximizes log p(θ | D) ∝ log p(D | θ) + log p(θ); it adds a prior regularization term to the MLE objective.

Question 18 Probability Distributions for ML

What is the REINFORCE estimator?

A score-function gradient estimator: ∇_θ E_{p_θ}[f(x)] = E_{p_θ}[f(x) ∇_θ log p_θ(x)]; unbiased but typically high-variance, reducible via baselines / control variates. MSE = (1/N) Σ (y_n − f(x_n))^2; it is the negative log-likelihood of a Gaussian with fixed variance, up to a constant. MLE returns x̄; MAP under a Gaussian prior N(μ_0, τ^2) returns (n τ^2 x̄ + σ^2 μ_0) / (n τ^2 + σ^2), a precision-weighted shrinkage estimate. A sequence of invertible, differentiable transformations f = f_K ∘ … ∘ f_1 mapping a simple base density (e.g. N(0, I)) to a complex one; log p(x) = log p_z(f^{-1}(x)) + Σ log |det(∂f_i^{−1}/∂x)|.

Question 19 Probability Distributions for ML

What is Hamiltonian Monte Carlo (HMC)?

An MCMC method that uses gradient information to propose distant moves that are accepted with high probability; effective for high-dimensional, smooth targets. GDA with the additional assumption that all classes share a common covariance Σ; the resulting Bayes decision boundary is linear in x. Continuous distribution with constant density f(x) = 1/(b−a) for x ∈ [a,b], else 0; mean (a+b)/2 and variance (b−a)^2/12. A non-informative prior proportional to √|I(θ)|, where I(θ) is the Fisher information; it is invariant to reparameterization.

Question 20 Probability Distributions for ML

What is an energy-based model (EBM)?

GDA without the equal-covariance assumption; the Bayes decision boundary is a quadratic function of x. A function (or rule) that assigns to every outcome in a sample space a non-negative probability, with the total probability over all outcomes summing (or integrating) to 1. Approximately 68% of mass within 1σ, 95% within 2σ, and 99.7% within 3σ of the mean. A model of the form p_θ(x) = exp(−E_θ(x)) / Z(θ) where E_θ is a learned energy function and Z(θ) = ∫ exp(−E_θ(x)) dx is a (typically intractable) normalizer.

Question 21 Probability Distributions for ML

What is the Weibull distribution?

A stochastic process where P(X_t | X_{1:t−1}) = P(X_t | X_{t−1}); the future is conditionally independent of the past given the present. An estimator f̂(x) = (1/(n h)) Σ_i K((x − x_i)/h) that places a smooth kernel K (bandwidth h) on each data point; an empirical approximation of the unknown density. A generalization of Exponential with shape k > 0 and scale λ: F(t) = 1 − exp(−(t/λ)^k) for t ≥ 0; used in reliability/survival analysis and for modeling time-to-failure. ∇_θ log p_θ(x); it is the gradient of the log-likelihood with respect to parameters and is the basis of REINFORCE and likelihood-ratio gradient estimators.

Question 22 Probability Distributions for ML

What is a reparameterized gradient estimator?

An estimator that writes x = g_θ(ε) with ε ~ q(ε) noise-free, so ∇_θ E_q[f(x)] = E_ε[∇_θ f(g_θ(ε))]; it has lower variance than score-function estimators in many settings. The distribution of sums (and more generally, sample covariances) of outer products of multivariate Gaussian vectors; it generalizes chi-squared to matrices and is the conjugate prior for the inverse covariance of a multivariate Normal. A multivariate Normal whose covariance Σ is singular (non-invertible); its mass lies on a lower-dimensional affine subspace and the PDF is undefined in the ambient space. A distribution whose tail decays slower than exponential (e.g. power-law P(X>x) ~ x^{−α}); it can produce extreme outliers and may have infinite variance.

Question 23 Probability Distributions for ML

What is the Maxwell-Boltzmann distribution?

A continuous distribution with ν degrees of freedom, mean 0 (for ν > 1) and variance ν/(ν−2) for ν > 2; it has heavier tails than the Normal and approaches Normal as ν → ∞. A chi distribution with 3 degrees of freedom; it describes the distribution of speeds of particles in an ideal gas. A multivariate generalization of Beta on the K-simplex: f(π) = (1/B(α)) Π π_i^{α_i−1} for π_i ≥ 0, Σ π_i = 1; it is the conjugate prior of the Multinomial. A distribution whose tail decays slower than exponential (e.g. power-law P(X>x) ~ x^{−α}); it can produce extreme outliers and may have infinite variance.

Question 24 Probability Distributions for ML

What is an isotropic Gaussian?

A tractable surrogate for score matching that corrupts data with Gaussian noise and matches the score of the noise-conditional model; basis of modern diffusion model training. A multivariate Normal with Σ = σ^2 I_d; its density depends only on Euclidean distance and is rotationally symmetric. It is the conjugate prior for the Bernoulli, Binomial, Geometric, and (with α,β > 1) provides a flexible distribution on probabilities. A generative model where each document has a topic-mixing vector θ_d ~ Dirichlet(α), each topic k has a word distribution φ_k ~ Dirichlet(β), and each word is drawn by choosing z ~ Categorical(θ_d) and w ~ Categorical(φ_z).

Question 25 Probability Distributions for ML

What is the Dirichlet-multinomial conjugacy?

A prior on correlation matrices indexed by a shape η ≥ 0; η = 1 is uniform on the space of correlation matrices and larger η concentrates density near the identity. Multinomial likelihood with Dirichlet(α) prior yields a Dirichlet(α + counts) posterior over category probabilities; this is foundational for Bayesian text models like LDA. VI that restricts q to a fully factorized family q(z) = Π_i q_i(z_i); leads to coordinate-ascent updates where each q_i is set to the log-normalized conditional expectation of log p under the others. An MCMC method that uses gradient information to propose distant moves that are accepted with high probability; effective for high-dimensional, smooth targets.

Question 26 Probability Distributions for ML

What is the 68-95-99.7 rule for the Normal?

The distribution of sums (and more generally, sample covariances) of outer products of multivariate Gaussian vectors; it generalizes chi-squared to matrices and is the conjugate prior for the inverse covariance of a multivariate Normal. A continuous distribution on (0,1) with PDF f(p) = p^{α−1} (1−p)^{β−1} / B(α,β), where B is the Beta function; mean α/(α+β). A stochastic process where P(X_t | X_{1:t−1}) = P(X_t | X_{t−1}); the future is conditionally independent of the past given the present. Approximately 68% of mass within 1σ, 95% within 2σ, and 99.7% within 3σ of the mean.

Question 27 Probability Distributions for ML

What is the connection between softmax and the Boltzmann distribution?

Both find a low-dimensional projection that maximizes class separability; Fisher's criterion gives a closed-form projection, while LDA arises from the generative model with shared covariance. Softmax(z) = exp(z_i)/Σ exp(z_j) is a Gibbs distribution over discrete states with "energy" −z_i and temperature 1; sampling from a softmax classifier with temperature T uses exp(z_i/T). F(x) = ∫_{-∞}^{x} f(t) dt and, where f is continuous, f(x) = dF/dx. A tractable surrogate for score matching that corrupts data with Gaussian noise and matches the score of the noise-conditional model; basis of modern diffusion model training.

Question 28 Probability Distributions for ML

What does the Mahalanobis distance measure?

A generalization of Bernoulli to K outcomes with probabilities p_1, …, p_K summing to 1; samples a single category. p(x) = ∫ p(x | θ) p(θ) dθ; the marginal distribution of data induced by the prior on parameters. d(x, μ) = √((x−μ)^T Σ^{−1} (x−μ)); the distance in units of standard deviations after whitening by Σ. A distribution whose PDF or PMF behaves as p(x) ∝ x^{−α} for large x; common in degrees of social networks, word frequencies (Zipf), and city sizes.

Question 29 Probability Distributions for ML

What is a Linear Discriminant Analysis (LDA) classifier?

The number of events in a fixed interval when events occur independently at a constant rate λ: P(X=k) = e^{−λ} λ^k / k!, with mean λ and variance λ. Approximately 68% of mass within 1σ, 95% within 2σ, and 99.7% within 3σ of the mean. A t-distribution with ν = 1 degree of freedom; PDF f(x) = 1/(π γ (1 + ((x−x0)/γ)^2)). It has undefined mean and variance and is a classic example of a heavy-tailed distribution. GDA with the additional assumption that all classes share a common covariance Σ; the resulting Bayes decision boundary is linear in x.

Question 30 Probability Distributions for ML

What is a latent variable?

Approximately 68% of mass within 1σ, 95% within 2σ, and 99.7% within 3σ of the mean. An unobserved random variable that influences observed data; in GMMs, the component assignment z is latent and learned via EM. softmax(z_i) = e^{z_i} / Σ_j e^{z_j} maps real-valued scores to a Categorical distribution; it is the multinomial logit / normalized exponential and arises naturally from the multinomial logistic regression likelihood. Multinomial likelihood with Dirichlet(α) prior yields a Dirichlet(α + counts) posterior over category probabilities; this is foundational for Bayesian text models like LDA.

Question 31 Probability Distributions for ML

What is a Geometric distribution (number-of-trials form)?

It is the conjugate prior for the Bernoulli, Binomial, Geometric, and (with α,β > 1) provides a flexible distribution on probabilities. softmax(z_i) = e^{z_i} / Σ_j e^{z_j} maps real-valued scores to a Categorical distribution; it is the multinomial logit / normalized exponential and arises naturally from the multinomial logistic regression likelihood. The number of trials until the first success in independent Bernoulli(p) trials: P(X=k) = (1−p)^{k−1} p for k = 1, 2, …, with mean 1/p. A distribution π such that π = πP; under irreducibility and aperiodicity, π exists, is unique, and is the long-run fraction of time the chain spends in each state.

Question 32 Probability Distributions for ML

What is a baseline (control variate) for variance reduction?

A linear model y = w^T x + ε with ε ~ N(0, σ^2) and Gaussian prior w ~ N(0, τ^2 I); posterior p(w | D) is Gaussian with closed-form mean (X^T X + (σ^2/τ^2) I)^{−1} X^T y. MSE = (1/N) Σ (y_n − f(x_n))^2; it is the negative log-likelihood of a Gaussian with fixed variance, up to a constant. p(x) = ∫ p(x | θ) p(θ) dθ; the marginal distribution of data induced by the prior on parameters. A function b(x) subtracted from f(x) in the REINFORCE estimator; valid because E[∇_θ log p_θ(x) · b(x)] = 0 when b does not depend on θ or sampling.

Question 33 Probability Distributions for ML

What is a Uniform distribution on [a, b]?

S(t) = P(T > t) = 1 − F(t); it gives the probability of "surviving" past time t and is central to survival analysis. Continuous distribution with constant density f(x) = 1/(b−a) for x ∈ [a,b], else 0; mean (a+b)/2 and variance (b−a)^2/12. d(x, μ) = √((x−μ)^T Σ^{−1} (x−μ)); the distance in units of standard deviations after whitening by Σ. A discrete power-law where the k-th most frequent item has probability ∝ 1/k^s; the basis of Zipf's law in word frequencies.

Question 34 Probability Distributions for ML

What is a diffusion model in distributional terms?

A model that learns to reverse a fixed noising Markov chain that gradually adds Gaussian noise to data; training minimizes a denoising score-matching loss on p(x_t | x_0) ~ N(√α̅_t x_0, (1−α̅_t) I). A distribution formed by drawing a component index z ~ Categorical(π) and then drawing x | z ~ p(x | z); the marginal density is p(x) = Σ_k π_k p_k(x). ELBO = E_q[log p(x,z)] + H(q) = log p(x) − D_KL(q(z) || p(z|x)); it is a tractable lower bound on the log evidence used in variational inference. E[X] = Σ_x x · p(x), assuming the sum converges; it is a measure of central tendency.

Question 35 Probability Distributions for ML

What is the kernel density estimator (KDE)?

A generalization of Binomial to K categories: counts (X_1,…,X_K) of n independent draws with Σ X_i = n and joint PMF n!/(Π x_i!) Π p_i^{x_i}. A continuous distribution with ν degrees of freedom, mean 0 (for ν > 1) and variance ν/(ν−2) for ν > 2; it has heavier tails than the Normal and approaches Normal as ν → ∞. An estimator f̂(x) = (1/(n h)) Σ_i K((x − x_i)/h) that places a smooth kernel K (bandwidth h) on each data point; an empirical approximation of the unknown density. Approximately 68% of mass within 1σ, 95% within 2σ, and 99.7% within 3σ of the mean.

Question 36 Probability Distributions for ML

What is Gibbs sampling?

σ(z) = 1/(1 + e^{−z}); it maps a real number to (0,1) and parameterizes Bernoulli probabilities in logistic regression. The square root of the variance: σ = √Var(X); it has the same units as X. An MCMC method that samples each variable from its full conditional given the others; a special case of Metropolis-Hastings with proposal equal to the conditional and acceptance 1. Continuous distribution with constant density f(x) = 1/(b−a) for x ∈ [a,b], else 0; mean (a+b)/2 and variance (b−a)^2/12.

Question 37 Probability Distributions for ML

What is the Central Limit Theorem (CLT)?

A continuous distribution with PDF f(x) = (1/(σ√(2π))) exp(−(x−μ)^2/(2σ^2)), parameterized by mean μ and variance σ^2. S(t) = P(T > t) = 1 − F(t); it gives the probability of "surviving" past time t and is central to survival analysis. The (properly normalized) sum of n i.i.d. random variables with finite mean μ and variance σ^2 converges in distribution to Normal(μ, σ^2/n) as n → ∞. Gamma(k=1, θ) is the Exponential(1/θ) distribution; Gamma models the sum of k i.i.d. Exponential(1/θ) waiting times.

Question 38 Probability Distributions for ML

What is a discrete probability distribution?

GDA with the additional assumption that all classes share a common covariance Σ; the resulting Bayes decision boundary is linear in x. A distribution defined on a countable set of outcomes, specified by a probability mass function (PMF) p(x) with p(x) ≥ 0 and Σ p(x) = 1. An estimator E_{p(x)}[f(x)] ≈ (1/N) Σ_i f(x_i) p(x_i)/q(x_i) with x_i ~ q; the choice of proposal q is critical for variance — it should have heavier tails than |f| p. A method that approximates an intractable posterior p(z|x) with a tractable q_φ(z) by minimizing D_KL(q_φ || p(·|x)), equivalently maximizing the ELBO.

Question 39 Probability Distributions for ML

What is the Beta distribution?

Var(X) = E[(X − E[X])^2] = E[X^2] − (E[X])^2; it measures the spread around the mean. A stochastic process where P(X_t | X_{1:t−1}) = P(X_t | X_{t−1}); the future is conditionally independent of the past given the present. A continuous distribution on (0,1) with PDF f(p) = p^{α−1} (1−p)^{β−1} / B(α,β), where B is the Beta function; mean α/(α+β). A function b(x) subtracted from f(x) in the REINFORCE estimator; valid because E[∇_θ log p_θ(x) · b(x)] = 0 when b does not depend on θ or sampling.

Question 40 Probability Distributions for ML

What is a Normal (Gaussian) distribution?

A prior p(θ) such that, when combined with a likelihood from a given family, the posterior p(θ | D) is in the same family as the prior (e.g. Beta–Bernoulli, Dirichlet–Multinomial, Normal–Normal, Gamma–Poisson, Inv-Wishart–Normal). Conjugate priors yield closed-form posteriors; for non-conjugate models, one resorts to MCMC, variational inference, or Laplace / EP approximations. F(x) = ∫_{-∞}^{x} f(t) dt and, where f is continuous, f(x) = dF/dx. A continuous distribution with PDF f(x) = (1/(σ√(2π))) exp(−(x−μ)^2/(2σ^2)), parameterized by mean μ and variance σ^2.

Question 41 Probability Distributions for ML

What is the relationship between Exponential and Gamma?

MLE returns x̄; MAP under a Gaussian prior N(μ_0, τ^2) returns (n τ^2 x̄ + σ^2 μ_0) / (n τ^2 + σ^2), a precision-weighted shrinkage estimate. An iterative algorithm: E-step computes posterior responsibilities γ_{nk} = π_k N(x_n; μ_k, Σ_k) / Σ_j π_j N(x_n; μ_j, Σ_j); M-step re-estimates π_k, μ_k, Σ_k from γ-weighted sufficient statistics. Gamma(k=1, θ) is the Exponential(1/θ) distribution; Gamma models the sum of k i.i.d. Exponential(1/θ) waiting times. MSE = (1/N) Σ (y_n − f(x_n))^2; it is the negative log-likelihood of a Gaussian with fixed variance, up to a constant.

Question 42 Probability Distributions for ML

What is the MAP principle?

A distribution whose tail decays slower than exponential (e.g. power-law P(X>x) ~ x^{−α}); it can produce extreme outliers and may have infinite variance. A t-distribution with ν = 1 degree of freedom; PDF f(x) = 1/(π γ (1 + ((x−x0)/γ)^2)). It has undefined mean and variance and is a classic example of a heavy-tailed distribution. Maximum a posteriori estimation maximizes log p(θ | D) ∝ log p(D | θ) + log p(θ); it adds a prior regularization term to the MLE objective. h(t) = f(t)/S(t), the instantaneous failure rate at time t conditional on survival to t; fully characterizes the distribution of a non-negative random variable together with S(0).

Question 43 Probability Distributions for ML

What is the MLE principle?

Maximum likelihood estimation chooses parameters θ to maximize the log-likelihood L(θ) = Σ_n log p(x_n | θ); equivalently it minimizes KL between the empirical distribution and p(· | θ) up to a constant. X = number of failures before the first success: P(X=k) = (1−p)^k p for k = 0, 1, 2, …, with mean (1−p)/p. An MCMC method that uses gradient information to propose distant moves that are accepted with high probability; effective for high-dimensional, smooth targets. A power-law with PDF f(x) = α x_m^α / x^{α+1} for x ≥ x_m; mean α x_m/(α−1) for α>1, variance finite only for α>2; widely used to model wealth and city size (the "80/20" rule).

Question 44 Probability Distributions for ML

What is a Gaussian Naive Bayes classifier?

A generative model assuming features are conditionally independent given class y, with P(x_i | y = c) = N(x_i; μ_{c,i}, σ²_{c,i}); decision rule is the maximum posterior class. π_i P_{ij} = π_j P_{ji} for all i, j; sufficient (with irreducibility) to guarantee that π is the stationary distribution. AIC = 2k − 2 log L̂; BIC = k log n − 2 log L̂; they penalize model complexity (k parameters) and BIC additionally penalizes more strongly with sample size n. A distribution whose PDF or PMF behaves as p(x) ∝ x^{−α} for large x; common in degrees of social networks, word frequencies (Zipf), and city sizes.

Question 45 Probability Distributions for ML

What is a Markov chain?

A continuous distribution with PDF f(x) = e^{−(x−μ)/s} / (s(1 + e^{−(x−μ)/s})^2), mean μ, variance s^2 π^2/3; its CDF is the logistic (sigmoid) function. A stochastic process where P(X_t | X_{1:t−1}) = P(X_t | X_{t−1}); the future is conditionally independent of the past given the present. A multivariate Normal with Σ = σ^2 I_d; its density depends only on Euclidean distance and is rotationally symmetric. p(x_new | D) = ∫ p(x_new | θ) p(θ | D) dθ; the Bayesian average of likelihoods over the posterior, used for prediction under uncertainty.

Question 46 Probability Distributions for ML

What is the Evidence Lower Bound (ELBO)?

A two-parameter family with shape k > 0 and rate θ > 0 (or scale β = 1/θ): PDF f(x) = x^{k−1} e^{−x/θ} / (Γ(k) θ^k) for x > 0; mean kθ, variance kθ^2. A distribution π such that π = πP; under irreducibility and aperiodicity, π exists, is unique, and is the long-run fraction of time the chain spends in each state. An MCMC method: propose y from q(y|x), accept with probability min(1, π(y) q(x|y) / (π(x) q(y|x))); the resulting chain has stationary distribution π. ELBO = E_q[log p(x,z)] + H(q) = log p(x) − D_KL(q(z) || p(z|x)); it is a tractable lower bound on the log evidence used in variational inference.

Question 47 Probability Distributions for ML

What is the difference between discriminative and generative models in distribution terms?

A two-layer undirected graphical model with visible units v and binary hidden units h; joint p(v, h) ∝ exp(b^T v + c^T h + v^T W h); trained by contrastive divergence. Discriminative models p(y|x) directly model the conditional; generative models model the joint p(x, y) or p(x), from which p(y|x) can be derived via Bayes' rule. D_KL(p || q) = E_{x∼p}[log p(x)/q(x)] ≥ 0; it measures how one distribution p differs from a reference q, but is asymmetric and not a true metric. The conjugate prior for a covariance matrix Σ; if Σ ~ Inv-Wishart(Ψ, ν) and data are i.i.d. multivariate Normal with known mean, the posterior is also inverse-Wishart.

Question 48 Probability Distributions for ML

What is the detailed balance condition?

For a convex function φ, φ(E[X]) ≤ E[φ(X)]; used to derive the ELBO and variational inference bounds. A positive random variable X whose log is Normal(μ, σ^2); its density is f(x) = (1/(xσ√(2π))) exp(−(ln x − μ)^2/(2σ^2)) for x > 0; mean e^{μ+σ^2/2}. The time between events in a Poisson process of rate λ: PDF f(t) = λ e^{−λt} for t ≥ 0; mean 1/λ, variance 1/λ^2, and it is memoryless. π_i P_{ij} = π_j P_{ji} for all i, j; sufficient (with irreducibility) to guarantee that π is the stationary distribution.

Question 49 Probability Distributions for ML

What is the Weibull hazard shape?

A method to generate two independent standard normal samples from two independent U(0,1) uniforms: Z0 = √(−2 ln U1) cos(2π U2), Z1 = √(−2 ln U1) sin(2π U2). MSE = (1/N) Σ (y_n − f(x_n))^2; it is the negative log-likelihood of a Gaussian with fixed variance, up to a constant. For shape k < 1 the hazard decreases (infant mortality), for k = 1 it is constant (Exponential), and for k > 1 it is increasing (wear-out). VI that restricts q to a fully factorized family q(z) = Π_i q_i(z_i); leads to coordinate-ascent updates where each q_i is set to the log-normalized conditional expectation of log p under the others.

Question 50 Probability Distributions for ML

What is a multivariate Gaussian (Normal) distribution?

X ∈ R^d with density f(x) = (2π)^{−d/2} |Σ|^{−1/2} exp(−½ (x−μ)^T Σ^{−1} (x−μ)); mean μ, covariance Σ (must be positive definite). Maximum a posteriori estimation maximizes log p(θ | D) ∝ log p(D | θ) + log p(θ); it adds a prior regularization term to the MLE objective. A prior p(θ) such that, when combined with a likelihood from a given family, the posterior p(θ | D) is in the same family as the prior (e.g. Beta–Bernoulli, Dirichlet–Multinomial, Normal–Normal, Gamma–Poisson, Inv-Wishart–Normal). A discrete power-law where the k-th most frequent item has probability ∝ 1/k^s; the basis of Zipf's law in word frequencies.