Title: AI Math — LearnCoachAssist

Q: What is a vector in the context of machine learning?
A: \[\mathbf{x} = \begin{bmatrix}x_1 \\ x_2 \\ \vdots \\ x_n\end{bmatrix} \in \mathbb{R}^n\]
Symbols: \(x_i\) = i-th component; \(\mathbb{R}^n\) = n-dimensional real space.
Intuition: A vector is an ordered list of n numbers. In ML it represents a data point, feature embedding, or parameter set as a point in n-dimensional space.

Q: How do you add two vectors?
A: \[\mathbf{a} + \mathbf{b} = \begin{bmatrix}a_1+b_1 \\ a_2+b_2 \\ \vdots \\ a_n+b_n\end{bmatrix}\]
Symbols: \(a_i, b_i\) = corresponding components of vectors a and b.
Intuition: Add element-wise. Geometrically, place the tail of b at the head of a; the sum points to the new head.

Q: How do you multiply a vector by a scalar?
A: \[c\mathbf{x} = \begin{bmatrix}cx_1 \\ cx_2 \\ \vdots \\ cx_n\end{bmatrix}\]
Symbols: \(c \in \mathbb{R}\) = scalar; \(x_i\) = i-th component.
Intuition: Scales every component by c. Stretches (|c|>1), shrinks (|c|<1), or flips direction (c<0) the vector.

Q: What is the L2 (Euclidean) norm of a vector?
A: \[\|\mathbf{x}\|_2 = \sqrt{\sum_{i=1}^n x_i^2}\]
Symbols: \(\|\cdot\|_2\) = L2 norm; \(x_i\) = i-th component; \(n\) = dimension.
Intuition: The straight-line (Euclidean) distance from the origin to the tip of the vector. Most common norm in ML for measuring distances.

Q: What is the L1 norm of a vector?
A: \[\|\mathbf{x}\|_1 = \sum_{i=1}^n |x_i|\]
Symbols: \(|x_i|\) = absolute value of i-th component.
Intuition: Sum of absolute values. Also called the 'Manhattan' or 'taxicab' distance. Encourages sparsity in optimization (L1 regularization).

Q: What is the general Lp norm of a vector?
A: \[\|\mathbf{x}\|_p = \left(\sum_{i=1}^n |x_i|^p\right)^{1/p}\]
Symbols: \(p \geq 1\) = norm order; \(x_i\) = i-th component.
Intuition: Generalizes L1 (p=1) and L2 (p=2). As \(p \to \infty\) it becomes the max norm \(\|\mathbf{x}\|_\infty = \max_i |x_i|\).

Q: What is the Frobenius norm of a matrix?
A: \[\|A\|_F = \sqrt{\sum_{i,j} A_{ij}^2}\]
Symbols: \(A_{ij}\) = element at row i, column j.
Intuition: The L2 norm applied to a matrix by treating it as a flattened vector. Equivalent to \(\sqrt{\mathrm{tr}(A^\top A)}\).

Q: What is the dot product of two vectors?
A: \[\mathbf{a} \cdot \mathbf{b} = \sum_{i=1}^n a_i b_i = \|\mathbf{a}\|_2\|\mathbf{b}\|_2\cos\theta\]
Symbols: \(a_i, b_i\) = components; \(\theta\) = angle between vectors.
Intuition: Measures how aligned two vectors are. Zero = orthogonal; positive = same direction; negative = opposite direction.

Q: How do you use the dot product to check orthogonality?
A: \[\mathbf{a} \perp \mathbf{b} \iff \mathbf{a} \cdot \mathbf{b} = 0\]
Symbols: \(\perp\) = perpendicular symbol.
Intuition: Two vectors are orthogonal (perpendicular) iff their dot product is zero — cos 90° = 0.

Q: How is the dot product used to compute the cosine similarity?
A: \[\cos\theta = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\|_2\|\mathbf{b}\|_2}\]
Symbols: \(\theta\) = angle between a and b.
Intuition: Normalizes the dot product by the product of norms to get a similarity score in [-1, 1]. Widely used in NLP to compare embedding vectors.

Q: What is the Cauchy-Schwarz inequality?
A: \[|\mathbf{a} \cdot \mathbf{b}| \leq \|\mathbf{a}\|_2 \|\mathbf{b}\|_2\]
Symbols: \(|\cdot|\) = absolute value; \(\|\cdot\|_2\) = L2 norm.
Intuition: The dot product can never exceed the product of the norms. Equality holds iff vectors are parallel.

Q: What is a matrix and how is it defined?
A: \[A \in \mathbb{R}^{m \times n},\quad A = \begin{bmatrix}a_{11} & \cdots & a_{1n} \\ \vdots & \ddots & \vdots \\ a_{m1} & \cdots & a_{mn}\end{bmatrix}\]
Symbols: \(m\) = rows; \(n\) = columns; \(a_{ij}\) = element at row i, col j.
Intuition: A rectangular array of numbers. In ML, represents linear maps, datasets (rows = samples, cols = features), or weight tensors.

Q: How do you add two matrices?
A: \[(A + B)_{ij} = A_{ij} + B_{ij}\]
Symbols: \(A_{ij}, B_{ij}\) = elements at position (i,j).
Intuition: Element-wise addition. Requires A and B to have the same shape \(m \times n\).

Q: How do you multiply a matrix by a scalar?
A: \[(cA)_{ij} = c \cdot A_{ij}\]
Symbols: \(c \in \mathbb{R}\) = scalar.
Intuition: Multiplies every element of the matrix by c. Scales the linear transformation represented by A.

Q: What is matrix multiplication and when is it defined?
A: \[C = AB,\quad C_{ij} = \sum_{k=1}^p A_{ik} B_{kj}\]
Symbols: \(A \in \mathbb{R}^{m \times p}\); \(B \in \mathbb{R}^{p \times n}\); \(C \in \mathbb{R}^{m \times n}\).
Intuition: Each entry C_ij is the dot product of row i of A with column j of B. Represents composing two linear transformations.

Q: Is matrix multiplication commutative?
A: \[AB \neq BA \text{ in general}\]
Symbols: \(A, B\) = square matrices (even for same shape).
Intuition: Unlike scalar multiplication, swapping order changes the result — order of linear transformations matters. Commutative property fails in general.

Q: What is the associativity property of matrix multiplication?
A: \[(AB)C = A(BC)\]
Symbols: \(A \in \mathbb{R}^{m \times n}\), \(B \in \mathbb{R}^{n \times p}\), \(C \in \mathbb{R}^{p \times q}\).
Intuition: You can group matrix products in any order. Crucial for efficient computation: choose the grouping that minimizes total operations.

Q: What is the matrix-vector product as a linear combination?
A: \[A\mathbf{x} = \sum_{j=1}^n x_j \mathbf{a}_j\]
Symbols: \(\mathbf{a}_j\) = j-th column of A; \(x_j\) = j-th component of x.
Intuition: Multiplying a matrix by a vector produces a linear combination of the matrix's columns, weighted by the vector's components.

Q: What is the transpose of a matrix?
A: \[(A^\top)_{ij} = A_{ji}\]
Symbols: \(A^\top\) = transpose of A; \(A \in \mathbb{R}^{m \times n}\), \(A^\top \in \mathbb{R}^{n \times m}\).
Intuition: Flips the matrix over its main diagonal — rows become columns and vice versa.

Q: What are the key properties of the matrix transpose?
A: \[(AB)^\top = B^\top A^\top, \quad (A^\top)^\top = A\]
Symbols: \(A, B\) = matrices; \(^\top\) = transpose.
Intuition: Transpose reverses the order of a product. This is analogous to how the inverse of a product reverses order: [(AB)^-1 = B^-1 A^-1].

Q: What is a symmetric matrix?
A: \[A = A^\top \iff A_{ij} = A_{ji} \text{ for all } i,j\]
Symbols: \(A^\top\) = transpose of A.
Intuition: A symmetric matrix equals its own transpose. Covariance matrices and Hessians are symmetric. Symmetric matrices always have real eigenvalues.

Q: What is the inverse of a square matrix?
A: \[AA^{-1} = A^{-1}A = I\]
Symbols: \(A^{-1}\) = inverse; \(I\) = identity matrix.
Intuition: The inverse 'undoes' the transformation of A. Exists only if A is square and full-rank (det(A) ≠ 0). Used to solve linear systems Ax = b.

Q: What is the inverse of a matrix product?
A: \[(AB)^{-1} = B^{-1}A^{-1}\]
Symbols: \(A, B\) = invertible matrices.
Intuition: To undo two successive transformations A then B, first undo B, then undo A — order reverses, just like the transpose.

Q: When does a matrix inverse exist?
A: \[A^{-1} \text{ exists} \iff \det(A) \neq 0 \iff \text{rank}(A) = n\]
Symbols: \(\det(A)\) = determinant; \(n\) = matrix dimension.
Intuition: A matrix is invertible if and only if it is full-rank — its columns are linearly independent and it doesn't squash space to a lower dimension.

Q: What is the inverse of an orthogonal matrix?
A: \[Q^{-1} = Q^\top \iff Q^\top Q = I\]
Symbols: \(Q\) = orthogonal matrix; \(I\) = identity.
Intuition: For orthogonal matrices, the inverse is simply the transpose — cheap to compute. These represent rotations/reflections that preserve lengths and angles.

Q: What is the identity matrix?
A: \[I_{ij} = \begin{cases}1 & i=j \\ 0 & i \neq j\end{cases},\quad AI = IA = A\]
Symbols: \(I \in \mathbb{R}^{n \times n}\) = identity matrix; 1s on diagonal, 0s off.
Intuition: The identity for matrix multiplication — leaves any matrix unchanged, like multiplying by 1 in scalar arithmetic.

Q: What is the zero matrix?
A: \[\mathbf{0} \in \mathbb{R}^{m \times n},\quad \mathbf{0}_{ij} = 0 \text{ for all } i,j\]
Symbols: All entries are zero.
Intuition: The additive identity: A + 0 = A. Any matrix times the zero matrix gives the zero matrix. A zero Hessian means no local curvature.

Q: What is the trace of a square matrix?
A: \[\mathrm{tr}(A) = \sum_{i=1}^n A_{ii}\]
Symbols: \(A_{ii}\) = diagonal entries; \(n\) = matrix size.
Intuition: Sum of diagonal elements. Equals the sum of all eigenvalues. Invariant under cyclic permutations of products.

Q: What is the cyclic property of the trace?
A: \[\mathrm{tr}(ABC) = \mathrm{tr}(BCA) = \mathrm{tr}(CAB)\]
Symbols: \(A, B, C\) = matrices with compatible dimensions.
Intuition: The trace is invariant under cyclic reordering. Useful in derivations: e.g., tr(xTAx) = tr(AxxT). Non-cyclic permutations are not equal in general.

Q: How does the trace relate to eigenvalues?
A: \[\mathrm{tr}(A) = \sum_{i=1}^n \lambda_i\]
Symbols: \(\lambda_i\) = eigenvalues of A.
Intuition: The trace is the sum of all eigenvalues (counting multiplicity). Together with the determinant (product of eigenvalues), these are key matrix invariants.

Q: What is the determinant of a 2×2 matrix?
A: \[\det\begin{pmatrix}a & b \\ c & d\end{pmatrix} = ad - bc\]
Symbols: \(a,b,c,d\) = matrix entries.
Intuition: For a 2×2 matrix, the signed area of the parallelogram formed by its column (or row) vectors. Zero determinant means the area collapsed — the matrix is singular.

Q: What is the geometric meaning of the determinant?
A: \[|\det(A)| = \text{volume scaling factor of linear map } A\]
Symbols: \(|\det(A)|\) = absolute value of determinant.
Intuition: The determinant tells how much the transformation A scales volumes. |det|=1: preserves volume; |det|=0: collapses to lower dimension; sign indicates orientation flip.

Q: What is the product rule for determinants?
A: \[\det(AB) = \det(A)\det(B)\]
Symbols: \(A, B \in \mathbb{R}^{n \times n}\).
Intuition: Volume scaling is multiplicative: applying A then B scales volume by det(A)·det(B). Therefore det(A^-1) = 1/det(A).

Q: How does the determinant relate to eigenvalues?
A: \[\det(A) = \prod_{i=1}^n \lambda_i\]
Symbols: \(\lambda_i\) = eigenvalues of A.
Intuition: The determinant equals the product of eigenvalues. If any eigenvalue is zero, det=0 and the matrix is singular. Key for understanding invertibility.

Q: What is the rank of a matrix?
A: \[\mathrm{rank}(A) = \dim(\mathrm{col}(A)) = \dim(\mathrm{row}(A))\]
Symbols: \(\mathrm{col}(A)\) = column space; \(\dim\) = dimension.
Intuition: The number of linearly independent columns (= rows). Equals the number of non-zero singular values. A full-rank matrix has rank = min(m,n).

Q: What is the rank-nullity theorem?
A: \[\mathrm{rank}(A) + \mathrm{nullity}(A) = n\]
Symbols: \(n\) = number of columns; \(\mathrm{nullity}(A) = \dim(\ker(A))\).
Intuition: The rank (number of dimensions A maps to) plus the nullity (dimensions collapsed to zero) always equals the number of input dimensions.

Q: What does it mean for a matrix to be rank-deficient?
A: \[\mathrm{rank}(A) < \min(m, n)\]
Symbols: \(m, n\) = rows and columns.
Intuition: Some columns are linear combinations of others — the matrix collapses information. Rank-deficient (singular) matrices have det=0 and are not invertible.

Q: What does it mean for vectors to be linearly independent?
A: \[c_1\mathbf{v}_1 + c_2\mathbf{v}_2 + \cdots + c_k\mathbf{v}_k = \mathbf{0} \implies c_1 = c_2 = \cdots = c_k = 0\]
Symbols: \(c_i \in \mathbb{R}\) = scalar coefficients; \(\mathbf{0}\) = zero vector.
Intuition: No vector in the set can be written as a combination of the others. Linearly independent vectors span the largest possible subspace.

Q: What is the span of a set of vectors?
A: \[\mathrm{span}\{\mathbf{v}_1,\ldots,\mathbf{v}_k\} = \left\{\sum_{i=1}^k c_i \mathbf{v}_i \,:\, c_i \in \mathbb{R}\right\}\]
Symbols: \(c_i\) = arbitrary scalars.
Intuition: All vectors reachable by linear combinations of \(\mathbf{v}_1,\ldots,\mathbf{v}_k\). The column space of a matrix is the span of its columns.

Q: What is a basis of a vector space?
A: \[\text{Basis} = \text{linearly independent set that spans the space}\]
Symbols: Any basis of \(\mathbb{R}^n\) has exactly n vectors.
Intuition: A minimal spanning set. Every vector in the space has a unique representation as a linear combination of basis vectors.

Q: What is the null space (kernel) of a matrix?
A: \[\ker(A) = \{\mathbf{x} \in \mathbb{R}^n : A\mathbf{x} = \mathbf{0}\}\]
Symbols: \(A \in \mathbb{R}^{m \times n}\); \(\mathbf{0}\) = zero vector.
Intuition: The set of all inputs A maps to zero — the 'invisible' directions. If the null space contains only the zero vector, A is injective (one-to-one).

Q: What is the column space of a matrix?
A: \[\mathrm{col}(A) = \{A\mathbf{x} : \mathbf{x} \in \mathbb{R}^n\} = \mathrm{span}\{\mathbf{a}_1,\ldots,\mathbf{a}_n\}\]
Symbols: \(\mathbf{a}_j\) = j-th column of A.
Intuition: All outputs reachable by A. The system Ax=b has a solution iff b lies in the column space of A.

Q: What is the row space of a matrix?
A: \[\mathrm{row}(A) = \mathrm{col}(A^\top)\]
Symbols: \(A^\top\) = transpose.
Intuition: The span of the rows of A. Orthogonal complement to the null space of A. dim(row space) = dim(col space) = rank(A).

Q: What is the eigenvalue equation?
A: \[A\mathbf{v} = \lambda\mathbf{v},\quad \mathbf{v} \neq \mathbf{0}\]
Symbols: \(A\) = square matrix; \(\lambda\) = eigenvalue (scalar); \(\mathbf{v}\) = eigenvector.
Intuition: An eigenvector is a special direction that A only stretches (by λ) but doesn't rotate. λ > 0: same direction; λ < 0: flipped; λ = 0: collapsed to zero.

Q: How do you find eigenvalues of a matrix?
A: \[\det(A - \lambda I) = 0\]
Symbols: \(\lambda\) = eigenvalue; \(I\) = identity matrix; \(\det\) = determinant.
Intuition: The characteristic polynomial. We look for λ values where A-λI is singular (non-invertible), meaning Ax=λx has non-trivial solutions.

Q: What is the characteristic polynomial?
A: \[p(\lambda) = \det(A - \lambda I) = (-1)^n\lambda^n + \cdots\]
Symbols: \(n\) = matrix dimension; polynomial roots = eigenvalues.
Intuition: An n×n matrix has exactly n eigenvalues (counting multiplicity, possibly complex). The polynomial encodes all eigenvalue information.

Q: What is the trace-eigenvalue and determinant-eigenvalue relationship?
A: \[\mathrm{tr}(A) = \sum_i \lambda_i, \quad \det(A) = \prod_i \lambda_i\]
Symbols: \(\lambda_i\) = eigenvalues.
Intuition: The trace gives the sum and the determinant gives the product of all eigenvalues. Quick checks: if any λ=0, det=0; if all λ>0, A is positive definite.

Q: Are eigenvectors corresponding to distinct eigenvalues linearly independent?
A: \[\lambda_i \neq \lambda_j \implies \mathbf{v}_i \not\parallel \mathbf{v}_j \text{ and } \{\mathbf{v}_i\} \text{ linearly independent}\]
Symbols: \(\lambda_i\) = distinct eigenvalues; \(\mathbf{v}_i\) = corresponding eigenvectors.
Intuition: Distinct eigenvalues guarantee linearly independent eigenvectors, which is necessary for diagonalization.

Q: What is the eigenspace corresponding to an eigenvalue λ?
A: \[E_\lambda = \ker(A - \lambda I) = \{\mathbf{v} : A\mathbf{v} = \lambda \mathbf{v}\}\]
Symbols: \(E_\lambda\) = eigenspace; \(\ker\) = null space.
Intuition: All eigenvectors for λ (plus the zero vector) form a subspace. Dimension of eigenspace = geometric multiplicity of λ.

Q: Do symmetric matrices always have real eigenvalues?
A: \[A = A^\top \implies \lambda_i \in \mathbb{R} \text{ and eigenvectors are orthogonal}\]
Symbols: \(A^\top\) = transpose; \(\lambda_i\) = eigenvalues.
Intuition: The spectral theorem: symmetric matrices are always diagonalizable with real eigenvalues and mutually orthogonal eigenvectors. Covariance matrices and Hessians benefit from this.

Q: What is eigendecomposition of a matrix?
A: \[A = Q\Lambda Q^{-1}\]
Symbols: \(Q\) = matrix of eigenvectors (columns); \(\Lambda = \mathrm{diag}(\lambda_1,\ldots,\lambda_n)\) = diagonal eigenvalue matrix.
Intuition: Factorizes A into: rotate to eigenbasis (Q^-1), scale each axis by eigenvalue (Λ), rotate back (Q). Possible only if A has n linearly independent eigenvectors.

Q: What is the eigendecomposition of a symmetric matrix?
A: \[A = Q\Lambda Q^\top, \quad Q^\top Q = I\]
Symbols: \(Q\) = orthogonal matrix of eigenvectors; \(\Lambda\) = diagonal matrix of eigenvalues.
Intuition: For symmetric matrices, Q is orthogonal (Q^-1 = Q^T) — eigenvectors are orthonormal. This is the spectral theorem and is the basis of PCA.

Q: How do you compute a matrix power using eigendecomposition?
A: \[A^k = Q\Lambda^k Q^{-1}, \quad \Lambda^k = \mathrm{diag}(\lambda_1^k,\ldots,\lambda_n^k)\]
Symbols: \(k\) = integer power.
Intuition: Powers of diagonal matrices are trivial — just raise each diagonal element to the power. So eigendecomposition makes matrix powers efficient.

Q: How do you compute the matrix exponential using eigendecomposition?
A: \[e^A = Q\, e^\Lambda\, Q^{-1}, \quad e^\Lambda = \mathrm{diag}(e^{\lambda_1},\ldots,e^{\lambda_n})\]
Symbols: \(e^A\) = matrix exponential; \(e^\Lambda\) = element-wise exponential of diagonal.
Intuition: Used in differential equations and continuous-time models. Works because in the eigenbasis, A acts independently on each axis.

Q: What is a positive definite matrix?
A: \[A \succ 0 \iff \mathbf{x}^\top A \mathbf{x} > 0 \text{ for all } \mathbf{x} \neq \mathbf{0}\]
Symbols: \(\succ 0\) = positive definite; \(\mathbf{x}^\top A\mathbf{x}\) = quadratic form.
Intuition: All eigenvalues are strictly positive. A bowl-shaped loss landscape is characterized by a positive definite Hessian — guaranteeing a unique minimum.

Q: What is a positive semi-definite (PSD) matrix?
A: \[A \succeq 0 \iff \mathbf{x}^\top A \mathbf{x} \geq 0 \text{ for all } \mathbf{x}\]
Symbols: \(\succeq 0\) = positive semi-definite.
Intuition: All eigenvalues are non-negative. Covariance matrices are always PSD — they may be singular (eigenvalue = 0) if features are perfectly correlated.

Q: How do you test if a matrix is positive definite using eigenvalues?
A: \[A \succ 0 \iff \lambda_i > 0 \text{ for all } i\]
Symbols: \(\lambda_i\) = eigenvalues of A.
Intuition: A symmetric matrix is positive definite iff all its eigenvalues are positive. Quick test: compute eigenvalues and check signs.

Q: What is Cholesky decomposition and when does it apply?
A: \[A = LL^\top, \quad L \text{ lower triangular with positive diagonal}\]
Symbols: \(L\) = lower-triangular Cholesky factor.
Intuition: A symmetric PSD matrix can be factored as LL^T. Used to efficiently solve linear systems and sample from multivariate Gaussians. Fails if A is not PSD.

Q: What is the Singular Value Decomposition (SVD)?
A: \[A = U\Sigma V^\top\]
Symbols: \(A \in \mathbb{R}^{m \times n}\); \(U \in \mathbb{R}^{m \times m}\) = left singular vectors (orthogonal); \(\Sigma \in \mathbb{R}^{m \times n}\) = diagonal (singular values \(\sigma_i \geq 0\)); \(V \in \mathbb{R}^{n \times n}\) = right singular vectors (orthogonal).
Intuition: Every matrix decomposes as: rotate (V^T), scale axes (Σ), rotate again (U). Generalizes eigendecomposition to non-square matrices.

Q: What are singular values and how do they relate to eigenvalues?
A: \[\sigma_i(A) = \sqrt{\lambda_i(A^\top A)} \geq 0\]
Symbols: \(\sigma_i\) = singular values sorted descending; \(\lambda_i(A^\top A)\) = eigenvalues of \(A^\top A\).
Intuition: Singular values measure how much A stretches space in each direction. They are always non-negative and generalize eigenvalues to rectangular matrices.

Q: How does SVD relate the rank of a matrix to its singular values?
A: \[\mathrm{rank}(A) = \#\{\sigma_i > 0\}\]
Symbols: \(\sigma_i\) = singular values.
Intuition: The rank equals the number of non-zero singular values. Singular values near zero indicate near-redundancy in the data — used for dimensionality assessment.

Q: What is the economy (thin) SVD?
A: \[A = U_r \Sigma_r V_r^\top, \quad r = \mathrm{rank}(A)\]
Symbols: \(U_r \in \mathbb{R}^{m \times r}\); \(\Sigma_r \in \mathbb{R}^{r \times r}\); \(V_r \in \mathbb{R}^{n \times r}\).
Intuition: Only keeps the r non-zero singular values/vectors. More compact than full SVD, sufficient for most computations.

Q: What is the connection between SVD and eigendecomposition of A^T A?
A: \[A^\top A = V\Sigma^\top U^\top U\Sigma V^\top = V\Sigma^2 V^\top\]
Symbols: \(V\) = right singular vectors; \(\Sigma^2\) = diagonal of squared singular values.
Intuition: The right singular vectors of A are eigenvectors of A^T A. Similarly, U are eigenvectors of AA^T. SVD and eigendecomposition are deeply connected.

Q: What is the best rank-k approximation of a matrix?
A: \[A_k = \sum_{i=1}^k \sigma_i \mathbf{u}_i \mathbf{v}_i^\top\]
Symbols: \(\sigma_i\) = i-th singular value; \(\mathbf{u}_i, \mathbf{v}_i\) = i-th left/right singular vectors.
Intuition: The Eckart-Young theorem: the best rank-k approximation (in Frobenius or spectral norm) is obtained by keeping only the top k singular triplets.

Q: What is the Eckart-Young theorem?
A: \[\min_{\mathrm{rank}(B)=k}\|A - B\|_F = \|A - A_k\|_F = \sqrt{\sum_{i=k+1}^r \sigma_i^2}\]
Symbols: \(A_k\) = rank-k SVD approximation; \(\sigma_i\) = singular values.
Intuition: Truncated SVD gives the closest rank-k matrix. Dropped singular values' squares sum to the squared reconstruction error. Basis of PCA and data compression.

Q: How is low-rank approximation used in recommendation systems?
A: \[\hat{R} \approx U_k \Sigma_k V_k^\top, \quad U_k, V_k \in \mathbb{R}^{*\times k}\]
Symbols: \(\hat{R}\) = predicted ratings matrix; \(k\) = latent factors.
Intuition: Factorizes user-item rating matrices into latent user and item factors. The k latent dimensions capture the most important patterns (genres, styles, etc.) in the data.

Q: What is Principal Component Analysis (PCA)?
A: \[\mathbf{z} = W^\top (\mathbf{x} - \boldsymbol{\mu}), \quad W = [\mathbf{u}_1, \ldots, \mathbf{u}_k]\]
Symbols: \(\mathbf{x}\) = data point; \(\boldsymbol{\mu}\) = mean; \(\mathbf{u}_i\) = i-th principal component (eigenvector of covariance matrix); \(k\) = reduced dimension.
Intuition: Projects data onto the k directions of maximum variance. Finds a low-dimensional subspace that retains as much information as possible.

Q: What is the covariance matrix in PCA and how is it computed?
A: \[\Sigma = \frac{1}{N-1}\sum_{i=1}^N (\mathbf{x}_i - \boldsymbol{\mu})(\mathbf{x}_i - \boldsymbol{\mu})^\top\]
Symbols: \(N\) = number of samples; \(\boldsymbol{\mu}\) = sample mean; \(\Sigma \in \mathbb{R}^{d \times d}\).
Intuition: Captures how each pair of features varies together. Symmetric and PSD. Its eigenvectors are the principal components; eigenvalues are the variances in those directions.

Q: How is explained variance computed in PCA?
A: \[\text{Explained variance ratio}_k = \frac{\sum_{i=1}^k \lambda_i}{\sum_{i=1}^d \lambda_i}\]
Symbols: \(\lambda_i\) = eigenvalues of the covariance matrix sorted descending; \(d\) = total features.
Intuition: The fraction of total variance captured by the top k components. Used to choose k: e.g., keep components explaining 95% of variance.

Q: How does PCA relate to SVD?
A: \[X = U\Sigma V^\top \implies \text{PCs} = V, \quad \Sigma^2/(N-1) = \Lambda\]
Symbols: \(X \in \mathbb{R}^{N \times d}\) = centered data matrix; \(V\) = right singular vectors = principal components.
Intuition: PCA via SVD of the centered data matrix is numerically more stable than eigendecomposing the covariance matrix, especially for high-dimensional data.

Q: What is the reconstruction formula in PCA?
A: \[\hat{\mathbf{x}} = W\mathbf{z} + \boldsymbol{\mu} = \sum_{i=1}^k z_i \mathbf{u}_i + \boldsymbol{\mu}\]
Symbols: \(\mathbf{z}\) = low-dimensional code; \(W\) = principal component matrix; \(\boldsymbol{\mu}\) = mean.
Intuition: Projects back from the compressed representation to the original space. Reconstruction error = variance of dropped components.

Q: What does it mean for a set of vectors to be orthonormal?
A: \[\mathbf{u}_i \cdot \mathbf{u}_j = \delta_{ij} = \begin{cases}1 & i=j \\ 0 & i \neq j\end{cases}\]
Symbols: \(\delta_{ij}\) = Kronecker delta; vectors are unit-length and mutually perpendicular.
Intuition: Orthonormal bases simplify computations dramatically — dot products become simple lookups, and projections are just dot products with basis vectors.

Q: What is the Gram-Schmidt process?
A: \[\mathbf{u}_k = \mathbf{v}_k - \sum_{j=1}^{k-1} \frac{\mathbf{v}_k \cdot \mathbf{u}_j}{\|\mathbf{u}_j\|^2}\mathbf{u}_j\]
Symbols: \(\mathbf{v}_k\) = k-th input vector; \(\mathbf{u}_j\) = previously computed orthogonal vectors.
Intuition: Iteratively removes from each new vector its projection onto all previous orthogonal vectors, leaving only the orthogonal component. Produces an orthogonal (or orthonormal) basis.

Q: What is QR decomposition?
A: \[A = QR, \quad Q^\top Q = I,\; R \text{ upper triangular}\]
Symbols: \(Q \in \mathbb{R}^{m \times n}\) = orthonormal columns; \(R \in \mathbb{R}^{n \times n}\) = upper triangular.
Intuition: Gram-Schmidt produces QR decomposition. Used to solve least-squares problems, compute eigenvalues, and numerically stable linear system solving.

Q: What is the projection of a vector onto another vector?
A: \[\mathrm{proj}_{\mathbf{u}}\mathbf{v} = \frac{\mathbf{v} \cdot \mathbf{u}}{\|\mathbf{u}\|^2}\mathbf{u}\]
Symbols: \(\mathbf{u}\) = direction to project onto; \(\mathbf{v} \cdot \mathbf{u}\) = dot product.
Intuition: The 'shadow' of v onto u. The scalar coefficient gives how much of v lies in the direction of u.

Q: What is the projection matrix onto the column space of A?
A: \[P = A(A^\top A)^{-1}A^\top\]
Symbols: \(A \in \mathbb{R}^{m \times n}\); \(P \in \mathbb{R}^{m \times m}\) = projection matrix; \(P^2 = P\) (idempotent).
Intuition: Projects any vector onto the column space of A. Fundamental in least squares: \(\hat{\mathbf{y}} = P\mathbf{y}\) is the closest point in col(A) to y.

Q: What is the least squares solution via projection?
A: \[\hat{\mathbf{x}} = (A^\top A)^{-1}A^\top \mathbf{b}\]
Symbols: \(A^\top A\) = Gram matrix; \(A^\top \mathbf{b}\) = normal equations right-hand side.
Intuition: Solves Ax ≈ b when no exact solution exists, by projecting b onto col(A). The normal equations \(A^\top A\hat{x} = A^\top b\) characterize the solution.

Q: What is an orthogonal projector and what are its properties?
A: \[P^2 = P, \quad P^\top = P, \quad \|P\mathbf{x}\| \leq \|\mathbf{x}\|\]
Symbols: \(P\) = orthogonal projection matrix; idempotent and symmetric.
Intuition: Projecting twice gives the same result (you're already in the subspace). Projections can only shrink or preserve vector length, never increase it.

Q: What is the covariance matrix of a random vector?
A: \[\Sigma = \mathbb{E}[(\mathbf{x}-\boldsymbol{\mu})(\mathbf{x}-\boldsymbol{\mu})^\top]\]
Symbols: \(\boldsymbol{\mu} = \mathbb{E}[\mathbf{x}]\) = mean vector; \(\Sigma_{ij} = \mathrm{Cov}(x_i, x_j)\).
Intuition: Encodes the variance of each feature (diagonal) and co-movement between features (off-diagonal). Always symmetric and positive semi-definite.

Q: What is the sample covariance matrix?
A: \[\hat{\Sigma} = \frac{1}{N-1}\sum_{i=1}^N (\mathbf{x}_i - \bar{\mathbf{x}})(\mathbf{x}_i - \bar{\mathbf{x}})^\top\]
Symbols: \(\bar{\mathbf{x}}\) = sample mean; \(N\) = number of samples; dividing by N-1 gives an unbiased estimator.
Intuition: Empirical estimate of the population covariance matrix. Positive semi-definite; singular if N-1 < d.

Q: What does a diagonal covariance matrix imply?
A: \[\Sigma = \mathrm{diag}(\sigma_1^2, \ldots, \sigma_d^2) \implies \text{features are uncorrelated}\]
Symbols: \(\sigma_i^2\) = variance of feature i; off-diagonal = 0.
Intuition: If all off-diagonal covariances are zero, features are uncorrelated (and independent if Gaussian). Naïve Bayes assumes this. PCA decorrelates features.

Q: What is the precision matrix?
A: \[\Lambda = \Sigma^{-1}\]
Symbols: \(\Lambda\) = precision (information) matrix; \(\Sigma\) = covariance matrix.
Intuition: The inverse of the covariance matrix. Its off-diagonal entries encode conditional independence (in Gaussian graphical models). Sparse precision = few direct dependencies.

Q: What is a quadratic form?
A: \[f(\mathbf{x}) = \mathbf{x}^\top A \mathbf{x} = \sum_{i,j} A_{ij} x_i x_j\]
Symbols: \(A \in \mathbb{R}^{n \times n}\) = symmetric matrix; \(\mathbf{x} \in \mathbb{R}^n\).
Intuition: A scalar function of a vector that is degree-2 in the components of x. The Hessian appears as A in the second-order Taylor expansion. A>0 means bowl-shaped.

Q: What is a change of basis?
A: \[[\mathbf{x}]_B = P^{-1}\mathbf{x}, \quad P = [\mathbf{b}_1 \mid \cdots \mid \mathbf{b}_n]\]
Symbols: \(P\) = change-of-basis matrix; \([\mathbf{x}]_B\) = coordinates of x in basis B.
Intuition: Represents the same vector in a different coordinate system. Eigendecomposition is a change of basis to the eigenvector basis where A acts diagonally.

Q: What is the Moore-Penrose pseudoinverse?
A: \[A^+ = V\Sigma^+ U^\top, \quad \Sigma^+_{ii} = \begin{cases}1/\sigma_i & \sigma_i \neq 0 \\ 0 & \sigma_i = 0\end{cases}\]
Symbols: \(A = U\Sigma V^\top\) = SVD; \(\Sigma^+\) = pseudoinverse of diagonal.
Intuition: Generalizes the matrix inverse to non-square or singular matrices. Gives the minimum-norm least-squares solution: \(\hat{\mathbf{x}} = A^+\mathbf{b}\).

Q: When does the pseudoinverse equal the regular inverse?
A: \[A \text{ invertible} \implies A^+ = A^{-1}\]
Symbols: \(A^+\) = pseudoinverse; \(A^{-1}\) = regular inverse.
Intuition: For full-rank square matrices the pseudoinverse reduces to the regular inverse. For rank-deficient or non-square matrices it provides the best possible approximate inverse.

Q: What is the minimum-norm least-squares solution using the pseudoinverse?
A: \[\hat{\mathbf{x}} = A^+\mathbf{b} = \arg\min_{\mathbf{x}} \|A\mathbf{x}-\mathbf{b}\|_2 \text{ with min } \|\mathbf{x}\|_2\]
Symbols: \(A^+\) = pseudoinverse; \(\mathbf{b}\) = target vector.
Intuition: Among all solutions that minimize the residual ‖Ax-b‖, picks the one with smallest norm. Key in underdetermined systems (more unknowns than equations).

Q: What is the spectral norm of a matrix?
A: \[\|A\|_2 = \sigma_{\max}(A)\]
Symbols: \(\sigma_{\max}\) = largest singular value of A.
Intuition: The maximum amount A stretches any unit vector. Used in spectral normalization (GANs) and stability analysis of neural networks.

Q: What is the nuclear norm of a matrix?
A: \[\|A\|_* = \sum_i \sigma_i\]
Symbols: \(\sigma_i\) = singular values of A.
Intuition: The sum of singular values. It is the convex relaxation of rank minimization — used in matrix completion and low-rank regularization, analogous to how L1 relaxes L0 for vectors.

Q: What is the Hadamard (element-wise) product of two matrices?
A: \[(A \odot B)_{ij} = A_{ij} \cdot B_{ij}\]
Symbols: \(\odot\) = Hadamard product; \(A, B \in \mathbb{R}^{m \times n}\) (same shape).
Intuition: Multiplies corresponding elements. Used in attention masks, dropout masks, and LSTM gate computations — contrasts with matrix multiplication which mixes rows and columns.

Q: What is the definition of a derivative?
A: \[f'(x) = \frac{df}{dx} = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h}\]
Symbols: \(h\) = infinitesimal increment; \(f'(x)\) = instantaneous rate of change.
Intuition: The slope of the tangent line to f at x. Tells how fast the output changes for a tiny change in input.

Q: What is the notation for a derivative?
A: \[f'(x) = \frac{df}{dx} = \frac{d}{dx}f(x) = \dot{f}\]
Symbols: Prime notation (Lagrange), Leibniz notation \(df/dx\), dot (Newton for time derivatives).
Intuition: All denote the same thing — rate of change of f with respect to x. In ML, Leibniz notation is common because it makes the variable explicit.

Q: What are the basic differentiation rules for power and constant functions?
A: \[\frac{d}{dx}[x^n] = nx^{n-1}, \quad \frac{d}{dx}[c] = 0\]
Symbols: \(n\) = exponent; \(c\) = constant.
Intuition: A constant has zero slope. A power function: drop the exponent as a coefficient, reduce exponent by 1. Foundation for polynomial differentiation.

Q: What is the sum rule for derivatives?
A: \[\frac{d}{dx}[f(x) + g(x)] = f'(x) + g'(x)\]
Symbols: \(f, g\) = differentiable functions.
Intuition: Derivative of a sum is the sum of derivatives. Derivatives are linear operators — this is the linearity property.

Q: What is the product rule for derivatives?
A: \[\frac{d}{dx}[f(x)g(x)] = f'(x)g(x) + f(x)g'(x)\]
Symbols: \(f, g\) = differentiable functions.
Intuition: 'First times derivative of second, plus second times derivative of first.' Arises because both factors change when x changes.

Q: What is the quotient rule for derivatives?
A: \[\frac{d}{dx}\left[\frac{f(x)}{g(x)}\right] = \frac{f'(x)g(x) - f(x)g'(x)}{[g(x)]^2}\]
Symbols: \(g(x) \neq 0\).
Intuition: 'Low d-high minus high d-low, over low squared.' Follows from product rule applied to f · (1/g).

Q: What is the chain rule for composite functions?
A: \[\frac{d}{dx}[f(g(x))] = f'(g(x)) \cdot g'(x)\]
Symbols: \(f \circ g\) = composite function; \(f'(g(x))\) = outer derivative evaluated at inner function.
Intuition: 'Derivative of outer times derivative of inner.' The most important rule in deep learning — backpropagation is essentially the chain rule applied repeatedly.

Q: How does the chain rule extend to multiple compositions?
A: \[\frac{d}{dx}[f_1(f_2(\cdots f_n(x)))] = f_1' \cdot f_2' \cdots f_n'\]
Symbols: Each \(f_k'\) is evaluated at the output of \(f_{k+1}(\cdots f_n(x))\).
Intuition: Multiply all the local slopes along the chain. In a deep neural network, this product of Jacobians propagates gradients from output to input.

Q: What is a partial derivative?
A: \[\frac{\partial f}{\partial x_i} = \lim_{h \to 0}\frac{f(x_1,\ldots,x_i+h,\ldots,x_n) - f(\mathbf{x})}{h}\]
Symbols: \(\partial\) = partial derivative symbol; all variables except \(x_i\) treated as constants.
Intuition: Rate of change of a multivariate function in one direction at a time. Forms the components of the gradient vector.

Q: What is the gradient vector of a scalar function?
A: \[\nabla f(\mathbf{x}) = \begin{bmatrix}\frac{\partial f}{\partial x_1} \\ \vdots \\ \frac{\partial f}{\partial x_n}\end{bmatrix}\]
Symbols: \(\nabla\) = nabla (gradient operator); \(\partial f / \partial x_i\) = i-th partial derivative.
Intuition: The vector of all partial derivatives. Points in the direction of steepest ascent of f. Its magnitude is the steepness. Used directly in gradient descent.

Q: What direction does the gradient point?
A: \[\nabla f(\mathbf{x}) \text{ points in the direction of steepest ascent of } f\]
Symbols: \(-\nabla f(\mathbf{x})\) = direction of steepest descent (used in gradient descent).
Intuition: If you stand on a hilly surface, the gradient points uphill. Moving in the negative gradient direction descends fastest — the foundation of gradient-based optimization.

Q: What is the gradient of a linear function f(x) = wᵀx?
A: \[\nabla_{\mathbf{x}} (\mathbf{w}^\top\mathbf{x}) = \mathbf{w}\]
Symbols: \(\mathbf{w}, \mathbf{x} \in \mathbb{R}^n\).
Intuition: The gradient of a linear function is just its coefficient vector. Constant gradient means constant slope in all directions — a flat (hyperplane) landscape.

Q: What is the gradient of a quadratic form f(x) = xᵀAx?
A: \[\nabla_{\mathbf{x}} (\mathbf{x}^\top A \mathbf{x}) = (A + A^\top)\mathbf{x}\]
Symbols: \(A \in \mathbb{R}^{n \times n}\); if A is symmetric: \(\nabla = 2A\mathbf{x}\).
Intuition: For the symmetric case (common in ML), gradient is 2Ax — analogous to the scalar derivative of ax² being 2ax. Key for understanding least squares.

Q: What is the directional derivative?
A: \[D_{\mathbf{u}} f(\mathbf{x}) = \nabla f(\mathbf{x}) \cdot \mathbf{u} = \|\nabla f\|\cos\theta\]
Symbols: \(\mathbf{u}\) = unit direction vector; \(\theta\) = angle between \(\nabla f\) and \(\mathbf{u}\).
Intuition: Rate of change of f in direction u. Maximum when u aligns with the gradient (θ=0); zero when u is perpendicular to gradient (θ=90°).

Q: What is the Jacobian matrix?
A: \[J_{ij} = \frac{\partial f_i}{\partial x_j}, \quad J \in \mathbb{R}^{m \times n}\]
Symbols: \(\mathbf{f}: \mathbb{R}^n \to \mathbb{R}^m\); \(J\) = matrix of all first-order partial derivatives.
Intuition: Generalizes the gradient to vector-valued functions. The Jacobian is the best linear approximation to f near a point — \(\mathbf{f}(\mathbf{x}+\delta) \approx \mathbf{f}(\mathbf{x}) + J\delta\).

Q: What is the shape of the Jacobian for a neural network layer output?
A: \[J = \frac{\partial \mathbf{a}}{\partial \mathbf{z}} \in \mathbb{R}^{m \times n}\]
Symbols: \(\mathbf{a} \in \mathbb{R}^m\) = layer output; \(\mathbf{z} \in \mathbb{R}^n\) = layer input.
Intuition: Rows correspond to output components, columns to input components. In backprop, we multiply Jacobians at each layer to propagate gradients.

Q: What is the determinant of the Jacobian (Jacobian determinant) used for?
A: \[\left|\det(J)\right| = \text{volume scaling factor of the transformation } \mathbf{f}\]
Symbols: \(J = \partial\mathbf{f}/\partial\mathbf{x}\); \(|\det J|\) = absolute Jacobian determinant.
Intuition: Appears in change-of-variable formulas for probability densities: \(p_x(\mathbf{x}) = p_z(\mathbf{f}(\mathbf{x}))|\det J|\). Central in normalizing flows.

Q: How is the multivariate chain rule expressed via Jacobians?
A: \[\frac{\partial \mathbf{f}}{\partial \mathbf{x}} = \frac{\partial \mathbf{f}}{\partial \mathbf{g}} \cdot \frac{\partial \mathbf{g}}{\partial \mathbf{x}}\]
Symbols: All terms are Jacobian matrices; product = matrix multiply.
Intuition: The chain rule generalizes to matrix multiplication of Jacobians. Backpropagation in neural networks is exactly this Jacobian chain rule, computed efficiently via reverse-mode autodiff.

Q: What is the Hessian matrix?
A: \[H_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j}\]
Symbols: \(H \in \mathbb{R}^{n \times n}\) for \(f: \mathbb{R}^n \to \mathbb{R}\); \(H = \nabla^2 f\).
Intuition: Matrix of all second partial derivatives. Captures the curvature of f. Symmetric (by Schwarz theorem). In optimization: tells whether a critical point is a min, max, or saddle.

Q: What does a positive definite Hessian imply at a critical point?
A: \[\nabla f(\mathbf{x}^*) = \mathbf{0} \text{ and } H \succ 0 \implies \mathbf{x}^* \text{ is a local minimum}\]
Symbols: \(H \succ 0\) = positive definite Hessian; all eigenvalues > 0.
Intuition: Positive definite curvature = bowl shape = local minimum. The loss landscape curves upward in every direction around the critical point.

Q: What does an indefinite Hessian imply at a critical point?
A: \[\nabla f(\mathbf{x}^*) = \mathbf{0} \text{ and } H \text{ indefinite} \implies \mathbf{x}^* \text{ is a saddle point}\]
Symbols: Indefinite: some eigenvalues > 0, some < 0.
Intuition: The surface curves up in some directions and down in others. Saddle points are common in deep networks — gradient is zero but it's neither a min nor max.

Q: How is the Hessian used in Newton's optimization method?
A: \[\mathbf{x}_{t+1} = \mathbf{x}_t - H^{-1}\nabla f(\mathbf{x}_t)\]
Symbols: \(H^{-1}\) = inverse Hessian; \(\nabla f\) = gradient.
Intuition: Uses curvature information to take a more direct step toward the minimum. Converges quadratically but computing H^-1 is O(n³) — impractical for large neural networks.

Q: What is the Taylor series expansion of a function?
A: \[f(x) = \sum_{k=0}^\infty \frac{f^{(k)}(a)}{k!}(x-a)^k\]
Symbols: \(f^{(k)}\) = k-th derivative; \(a\) = expansion point; \(k!\) = factorial.
Intuition: Any smooth function can be written as an infinite polynomial around any point. In ML, we mostly use the first two terms for local approximations.

Q: What is the first-order (linear) Taylor approximation?
A: \[f(\mathbf{x} + \boldsymbol{\delta}) \approx f(\mathbf{x}) + \nabla f(\mathbf{x})^\top \boldsymbol{\delta}\]
Symbols: \(\boldsymbol{\delta}\) = small perturbation vector; \(\nabla f(\mathbf{x})\) = gradient at x.
Intuition: The function changes by approximately the dot product of the gradient with the perturbation. This is why gradient descent works for small steps.

Q: What is the second-order Taylor approximation?
A: \[f(\mathbf{x} + \boldsymbol{\delta}) \approx f(\mathbf{x}) + \nabla f^\top\boldsymbol{\delta} + \frac{1}{2}\boldsymbol{\delta}^\top H\boldsymbol{\delta}\]
Symbols: \(H\) = Hessian matrix; \(\boldsymbol{\delta}\) = perturbation.
Intuition: Adds a curvature correction term ½δᵀHδ. Positive definite H means this term resists large steps. Basis for Newton's method and second-order optimizers.

Q: What is the first-order Taylor approximation used for in gradient descent analysis?
A: \[\mathcal{L}(\theta - \eta\nabla\mathcal{L}) \approx \mathcal{L}(\theta) - \eta\|\nabla\mathcal{L}\|^2\]
Symbols: \(\eta\) = learning rate; \(\|\nabla\mathcal{L}\|^2\) = squared gradient norm.
Intuition: Each gradient descent step decreases the loss by approximately η‖∇L‖². Valid only for small η (otherwise the linear approx. breaks down).

Q: What is a critical point?
A: \[\nabla f(\mathbf{x}^*) = \mathbf{0}\]
Symbols: \(\mathbf{x}^*\) = critical point; all partial derivatives are zero.
Intuition: A point where the gradient vanishes — could be a minimum, maximum, or saddle point. Gradient descent converges to critical points but not necessarily to global minima.

Q: What is a saddle point and why is it relevant to deep learning?
A: \[\nabla f(\mathbf{x}^*) = \mathbf{0},\quad \exists\, \lambda_i > 0,\, \exists\, \lambda_j < 0 \text{ in Hessian}\]
Symbols: \(\lambda_i, \lambda_j\) = eigenvalues of the Hessian at \(\mathbf{x}^*\).
Intuition: Gradient is zero but the point is neither a min nor max. Common in high-dimensional loss landscapes. SGD with noise often escapes saddle points that pure GD might slow near.

Q: What is a convex function?
A: \[f(\lambda\mathbf{x} + (1-\lambda)\mathbf{y}) \leq \lambda f(\mathbf{x}) + (1-\lambda)f(\mathbf{y}), \quad \lambda \in [0,1]\]
Symbols: \(\lambda\) = interpolation parameter; \(\mathbf{x}, \mathbf{y}\) = any two points.
Intuition: The chord between any two points lies above the function. Convex functions have no local minima that aren't global — gradient descent is guaranteed to converge.

Q: What is the second-order condition for convexity?
A: \[f \text{ convex} \iff \nabla^2 f(\mathbf{x}) \succeq 0 \text{ for all } \mathbf{x}\]
Symbols: \(\nabla^2 f\) = Hessian; \(\succeq 0\) = positive semi-definite.
Intuition: A function is convex iff its Hessian is PSD everywhere — non-negative curvature in all directions. Strictly convex has PD Hessian and a unique global minimum.

Q: What is Jensen's inequality for convex functions?
A: \[f\!\left(\mathbb{E}[X]\right) \leq \mathbb{E}[f(X)]\]
Symbols: \(X\) = random variable; \(\mathbb{E}\) = expectation.
Intuition: For convex f, the function of the mean ≤ the mean of the function. Used in proving KL divergence ≥ 0, EM algorithm bounds, and many information-theoretic inequalities.

Q: What is the total derivative of f(x,y) where x and y both depend on t?
A: \[\frac{df}{dt} = \frac{\partial f}{\partial x}\frac{dx}{dt} + \frac{\partial f}{\partial y}\frac{dy}{dt}\]
Symbols: \(df/dt\) = total rate of change; \(\partial f/\partial x\) = partial derivatives.
Intuition: Both x and y change with t, so f changes through both pathways. Sum contributions from all paths — the generalization of the chain rule to multiple variables.

Q: What is the multivariate chain rule in matrix form?
A: \[\frac{\partial z}{\partial \mathbf{x}} = \frac{\partial z}{\partial \mathbf{y}} \cdot \frac{\partial \mathbf{y}}{\partial \mathbf{x}}\]
Symbols: \(\partial z/\partial \mathbf{x}\) = row gradient; \(\partial \mathbf{y}/\partial \mathbf{x}\) = Jacobian matrix.
Intuition: Chain rule as matrix product. In backpropagation, this is computed right-to-left (reverse mode) because it's cheaper to multiply row-vector by matrix than matrix by matrix.

Q: What is implicit differentiation?
A: \[F(x,y) = 0 \implies \frac{dy}{dx} = -\frac{\partial F/\partial x}{\partial F/\partial y}\]
Symbols: \(F(x,y)\) = implicit equation defining y as a function of x.
Intuition: When y is defined implicitly (not explicitly as y=f(x)), differentiate both sides with respect to x and solve for dy/dx. Used in meta-learning and implicit models.

Q: What are Lagrange multipliers?
A: \[\nabla f(\mathbf{x}^*) = \lambda \nabla g(\mathbf{x}^*), \quad g(\mathbf{x}^*) = 0\]
Symbols: \(\lambda\) = Lagrange multiplier; \(g(\mathbf{x})=0\) = equality constraint.
Intuition: At a constrained optimum, the gradients of objective and constraint are parallel — f can't improve without violating the constraint. λ measures the 'exchange rate' between objective and constraint.

Q: What is the Lagrangian function for constrained optimization?
A: \[\mathcal{L}(\mathbf{x}, \lambda) = f(\mathbf{x}) + \lambda g(\mathbf{x})\]
Symbols: \(f\) = objective; \(g(\mathbf{x})=0\) = equality constraint; \(\lambda\) = multiplier.
Intuition: Transforms a constrained problem into an unconstrained one. Setting ∇L=0 w.r.t. both x and λ gives the stationarity and feasibility conditions.

Q: What are the KKT conditions for inequality constraints?
A: \[\nabla f + \sum_i \mu_i \nabla h_i = 0, \quad h_i(\mathbf{x}) \leq 0, \quad \mu_i \geq 0, \quad \mu_i h_i = 0\]
Symbols: \(\mu_i \geq 0\) = inequality multipliers; \(h_i \leq 0\) = inequality constraints; \(\mu_i h_i=0\) = complementary slackness.
Intuition: KKT generalizes Lagrange multipliers to inequality constraints. Complementary slackness: either the constraint is active (h=0) or its multiplier is zero (constraint doesn't bind).

Q: What is complementary slackness in KKT conditions?
A: \[\mu_i h_i(\mathbf{x}^*) = 0 \text{ for all } i\]
Symbols: \(\mu_i\) = KKT multiplier; \(h_i\) = i-th inequality constraint.
Intuition: Either the constraint is tight (h_i=0, constraint is active) or the multiplier is zero (μ_i=0, constraint is inactive and doesn't affect the solution). Not both can be positive.

Q: What is the Leibniz integral rule (differentiation under the integral sign)?
A: \[\frac{d}{d\theta}\int_{a(\theta)}^{b(\theta)} f(x,\theta)\,dx = \int_{a}^{b} \frac{\partial f}{\partial \theta}dx + f(b,\theta)b'(\theta) - f(a,\theta)a'(\theta)\]
Symbols: \(a(\theta), b(\theta)\) = limits depending on θ.
Intuition: Swapping derivative and integral. In ML (fixed limits): \(\frac{d}{d\theta}\mathbb{E}_p[f] = \mathbb{E}_p\left[\frac{\partial f}{\partial\theta}\right]\) — key for score function estimators.

Q: What is the reparameterization trick in variational inference?
A: \[\mathbf{z} = \mu + \sigma \odot \boldsymbol{\epsilon}, \quad \boldsymbol{\epsilon} \sim \mathcal{N}(0, I)\]
Symbols: \(\mu, \sigma\) = learned mean/std; \(\odot\) = element-wise multiply; \(\boldsymbol{\epsilon}\) = noise sample.
Intuition: Makes the sampling operation differentiable by moving the stochasticity to ε (which doesn't depend on parameters). Enables backprop through stochastic nodes in VAEs.

Q: What is integration by parts?
A: \[\int u\,dv = uv - \int v\,du\]
Symbols: \(u, v\) = functions; \(du = u'dx\), \(dv = v'dx\).
Intuition: Reverse of the product rule for differentiation. Used to derive score function gradients and in the analysis of entropy/KL divergence.

Q: What is the score function (REINFORCE) gradient estimator?
A: \[\nabla_\theta \mathbb{E}_{p_\theta}[f(x)] = \mathbb{E}_{p_\theta}\left[f(x)\nabla_\theta \log p_\theta(x)\right]\]
Symbols: \(\nabla_\theta \log p_\theta(x)\) = score function; \(f(x)\) = reward/objective.
Intuition: Uses the log-derivative trick to compute gradient of expectation. Used in REINFORCE (RL) and when reparameterization is not possible (discrete latents).

Q: What is the derivative of the exponential function?
A: \[\frac{d}{dx}e^x = e^x, \quad \frac{d}{dx}e^{ax} = ae^{ax}\]
Symbols: \(a\) = constant; \(e\) = Euler's number ≈ 2.718.
Intuition: The exponential function is its own derivative — it grows at a rate equal to its current value. Critical property underlying all activation functions involving exp.

Q: What is the derivative of the natural logarithm?
A: \[\frac{d}{dx}\ln(x) = \frac{1}{x}, \quad x > 0\]
Symbols: \(\ln\) = natural logarithm (base e).
Intuition: Logarithm grows slowly but its derivative is large near 0 (fast initial change). Cross-entropy loss contains log — its gradient is 1/(predicted probability).

Q: What is the derivative of the sigmoid function?
A: \[\frac{d}{dx}\sigma(x) = \sigma(x)(1-\sigma(x))\]
Symbols: \(\sigma(x) = 1/(1+e^{-x})\) = sigmoid function.
Intuition: Elegant self-referential formula. Maximum at x=0 (value 0.25). Vanishes for large |x| (saturation) — this causes vanishing gradients in deep sigmoid networks.

Q: What is the derivative of the softmax function?
A: \[\frac{\partial \sigma_i}{\partial z_j} = \sigma_i(\delta_{ij} - \sigma_j)\]
Symbols: \(\sigma_i = e^{z_i}/\sum_k e^{z_k}\); \(\delta_{ij}\) = Kronecker delta.
Intuition: The Jacobian of softmax is not diagonal — changing z_j affects all outputs. The δ_ij term handles i=j; the -σ_j term captures the normalization interaction.

Q: What is the derivative of tanh?
A: \[\frac{d}{dx}\tanh(x) = 1 - \tanh^2(x) = \mathrm{sech}^2(x)\]
Symbols: \(\tanh(x) = (e^x-e^{-x})/(e^x+e^{-x})\); \(\mathrm{sech}\) = hyperbolic secant.
Intuition: Max derivative = 1 at x=0; vanishes near saturation. Better than sigmoid (zero-centered output) but still saturates. Used in LSTMs and RNNs.

Q: What is L'Hôpital's rule?
A: \[\lim_{x \to a}\frac{f(x)}{g(x)} = \lim_{x \to a}\frac{f'(x)}{g'(x)}, \quad \text{when form is } \frac{0}{0} \text{ or } \frac{\infty}{\infty}\]
Symbols: \(f'(x), g'(x)\) = derivatives of numerator and denominator.
Intuition: When a ratio has indeterminate form, replace by ratio of derivatives. Used in analyzing limits of loss functions, e.g., \(\lim_{p\to 0} p\log p = 0\) in entropy.

Q: What is the mean value theorem?
A: \[\exists\, c \in (a,b): f'(c) = \frac{f(b)-f(a)}{b-a}\]
Symbols: \(c\) = some point between a and b; the right side = average slope.
Intuition: There exists some point where the instantaneous slope equals the average slope over the interval. Foundation for gradient-based convergence proofs.

Q: What is the relationship between differentiability and continuity?
A: \[f \text{ differentiable at } x \implies f \text{ continuous at } x\]
Symbols: The converse is false — e.g., \(|x|\) is continuous but not differentiable at 0.
Intuition: Differentiability is stronger than continuity. ReLU is continuous everywhere but not differentiable at 0 — we handle this with subgradients.

Q: What is a Lipschitz continuous function?
A: \[|f(x) - f(y)| \leq L|x - y| \text{ for all } x, y\]
Symbols: \(L\) = Lipschitz constant; bounds the maximum slope of f.
Intuition: The function doesn't change too fast — slope is bounded by L. Used in convergence analysis: gradient descent with step size η < 1/L is guaranteed to converge for L-smooth convex functions.

Q: What is a subgradient for non-differentiable functions?
A: \[f(\mathbf{y}) \geq f(\mathbf{x}) + \mathbf{g}^\top(\mathbf{y}-\mathbf{x}) \text{ for all } \mathbf{y}\]
Symbols: \(\mathbf{g}\) = subgradient at x; set of all subgradients = subdifferential \(\partial f(\mathbf{x})\).
Intuition: Generalizes gradient to non-differentiable functions. At a kink (like ReLU at 0 or |x| at 0), any slope between the left and right limits is a valid subgradient.

Q: What is the subdifferential of the absolute value function |x|?
A: \[\partial |x| = \begin{cases}\{1\} & x > 0 \\ [-1,1] & x = 0 \\ \{-1\} & x < 0\end{cases}\]
Symbols: \(\partial |x|\) = set of all subgradients at x.
Intuition: At the non-smooth point x=0, any value in [-1,1] is a valid subgradient. L1 regularization uses |w|, and its subgradient at 0 being an interval is what allows sparsity.

Q: What is forward-mode automatic differentiation?
A: \[\dot{y} = \frac{\partial y}{\partial x} \cdot \dot{x}, \quad \dot{x} \text{ set to 1}\]
Symbols: \(\dot{x}\) = tangent (derivative of x w.r.t. seed input); propagated forward through computation.
Intuition: Computes one directional derivative in one forward pass. Efficient when inputs << outputs (e.g., Jacobian columns). Implemented via dual numbers.

Q: What is reverse-mode automatic differentiation (backpropagation)?
A: \[\bar{x}_i = \sum_j \bar{y}_j \frac{\partial y_j}{\partial x_i}\]
Symbols: \(\bar{x}_i = \partial L/\partial x_i\) = adjoint (gradient flowing back); \(\bar{y}_j = \partial L/\partial y_j\) = gradient from output.
Intuition: One backward pass computes gradients w.r.t. all inputs simultaneously. Efficient when outputs << inputs (one scalar loss, many parameters). Used in all DL frameworks.

Q: Why is reverse-mode autodiff preferred for training neural networks?
A: \[\text{Cost} \propto O(n_{\text{output}}), \text{ not } O(n_{\text{input}})\]
Symbols: \(n_{\text{output}} = 1\) (scalar loss); \(n_{\text{input}}\) = millions of parameters.
Intuition: With one scalar loss and millions of parameters, reverse mode computes all gradients in one backward pass at approximately the cost of one forward pass — far cheaper than forward mode for training.

Q: What is the gradient of a squared L2 norm?
A: \[\nabla_{\mathbf{x}} \|\mathbf{x}\|_2^2 = 2\mathbf{x}\]
Symbols: \(\|\mathbf{x}\|_2^2 = \mathbf{x}^\top\mathbf{x}\).
Intuition: Analogous to d/dx[x²] = 2x for scalars. Appears constantly in L2 regularization gradients and least-squares derivations.

Q: What is the gradient of the loss w.r.t. weights in linear regression?
A: \[\nabla_W \mathcal{L} = \nabla_W \|Xw - y\|^2 = 2X^\top(Xw - y)\]
Symbols: \(X \in \mathbb{R}^{N \times d}\) = data; \(w\) = weights; \(y\) = targets.
Intuition: The gradient of MSE loss: proportional to the input transposed times the residual vector. Setting to zero gives the normal equations.

Q: What is the gradient of the cross-entropy loss with respect to logits?
A: \[\frac{\partial \mathcal{L}}{\partial z_i} = \hat{y}_i - y_i\]
Symbols: \(\hat{y}_i = \sigma(z_i)\) = softmax probability; \(y_i\) = true label (one-hot); \(z_i\) = logit.
Intuition: The gradient of cross-entropy + softmax w.r.t. logits is simply the prediction minus the label — a beautifully clean formula.

Q: How is the chain rule applied in backpropagation for a single layer?
A: \[\frac{\partial \mathcal{L}}{\partial W} = \frac{\partial \mathcal{L}}{\partial z} \cdot \frac{\partial z}{\partial W}, \quad z = Wx + b\]
Symbols: \(\mathcal{L}\) = loss; \(z\) = pre-activation; \(W\) = weight matrix.
Intuition: Chain rule connects the upstream gradient ∂L/∂z to the weight gradient. For a linear layer: ∂L/∂W = (∂L/∂z)^T · x^T.

Q: What is the gradient of a composition f(g(x)) where g is a vector function?
A: \[\frac{\partial f}{\partial \mathbf{x}} = \frac{\partial f}{\partial \mathbf{g}} J_g\]
Symbols: \(J_g = \partial\mathbf{g}/\partial\mathbf{x}\) = Jacobian of g; row-vector times Jacobian.
Intuition: The upstream gradient is left-multiplied by the Jacobian of the inner function. This is the vector-chain-rule used in every DL backward pass.

Q: What is the condition number of the Hessian and why does it matter?
A: \[\kappa(H) = \frac{\lambda_{\max}}{\lambda_{\min}}\]
Symbols: \(\lambda_{\max}, \lambda_{\min}\) = largest and smallest eigenvalues of H.
Intuition: High condition number means the loss is much more curved in some directions than others — a narrow valley. Gradient descent oscillates in high-condition-number landscapes; adaptive optimizers (Adam) implicitly address this.

Q: What is the gradient clipping rule?
A: \[\text{if } \|\mathbf{g}\| > \tau: \quad \mathbf{g} \leftarrow \frac{\tau}{\|\mathbf{g}\|}\mathbf{g}\]
Symbols: \(\mathbf{g}\) = gradient vector; \(\tau\) = clipping threshold; \(\|\mathbf{g}\|\) = gradient norm.
Intuition: When gradients explode (huge norms), scale them back to have norm τ while preserving direction. Essential for training RNNs and transformers on long sequences.

Q: What is the zeroth-order (constant) Taylor approximation?
A: \[f(\mathbf{x} + \boldsymbol{\delta}) \approx f(\mathbf{x})\]
Symbols: \(\boldsymbol{\delta}\) = perturbation (assumed tiny).
Intuition: Zero-order: just use the current value. Used in zero-order optimization (black-box optimization) where gradients are unavailable — only function evaluations.

Q: What is the Taylor expansion of the softplus function?
A: \[\text{softplus}(x) = \ln(1 + e^x) \approx x \text{ for large } x, \approx e^x \text{ for small } x\]
Symbols: Smooth approximation to ReLU.
Intuition: Softplus is the smooth upper envelope of ReLU. Its derivative is the sigmoid: d/dx[softplus(x)] = σ(x). Taylor expansion helps analyze behavior in different regimes.

Q: What is the chain rule for the log function in ML?
A: \[\frac{\partial}{\partial\theta}\log p(x|\theta) = \frac{1}{p(x|\theta)}\frac{\partial p(x|\theta)}{\partial\theta}\]
Symbols: \(p(x|\theta)\) = likelihood; \(\partial p/\partial\theta\) = gradient of probability w.r.t. parameters.
Intuition: The log-derivative trick. MLE training maximizes log-likelihood — gradients involve 1/p times ∂p/∂θ, which is more numerically stable than working with p directly.

Q: What is the gradient of the log-sum-exp function?
A: \[\nabla_\mathbf{z}\,\log\sum_i e^{z_i} = \mathrm{softmax}(\mathbf{z})\]
Symbols: \(z_i\) = logits; softmax gives the partial derivatives.
Intuition: LogSumExp is the smooth maximum function. Its gradient is exactly the softmax probabilities — this is why softmax + cross-entropy has such clean gradients.

Q: What is the partial derivative of the L2 norm with respect to a vector component?
A: \[\frac{\partial}{\partial x_i}\|\mathbf{x}\|_2 = \frac{x_i}{\|\mathbf{x}\|_2}\]
Symbols: \(\|\mathbf{x}\|_2 = \sqrt{\sum_j x_j^2}\); gradient = \(\mathbf{x}/\|\mathbf{x}\|_2\) = unit vector.
Intuition: The gradient of the L2 norm points in the direction of x (unit vector). Used in gradient clipping, weight normalization, and cosine similarity gradient computations.

Q: What is automatic differentiation vs symbolic differentiation?
A: \[\text{Autodiff: evaluate } \frac{\partial L}{\partial \theta} \text{ numerically; Symbolic: derive closed-form expression}\]
Symbols: Autodiff operates on computational graphs; symbolic on math expressions.
Intuition: Autodiff (used by PyTorch/JAX) applies chain rule numerically during execution — no expression swell. Symbolic differentiation produces exact formulas but can blow up in size.

Q: What is the finite difference approximation to a derivative?
A: \[f'(x) \approx \frac{f(x+h) - f(x)}{h} \approx \frac{f(x+h) - f(x-h)}{2h}\]
Symbols: \(h\) = small step size; centered difference (second formula) is more accurate O(h²).
Intuition: Numerical gradient estimation. Used for gradient checking in implementations. Too large h = inaccurate; too small h = floating point errors. Centered differences halve the error.

Q: What is the gradient tape / computational graph in automatic differentiation?
A: \[\text{Forward: } y = f(x) \text{ records ops; Backward: } \frac{\partial L}{\partial x} = \frac{\partial L}{\partial y}\frac{\partial y}{\partial x}\]
Symbols: Computational graph = DAG of operations; tape = recorded forward pass.
Intuition: During the forward pass, a 'tape' records all operations. During backprop, it replays the tape in reverse to compute all gradients via the chain rule.

Q: What is the derivative of the ReLU function?
A: \[\frac{d}{dx}\text{ReLU}(x) = \begin{cases}1 & x > 0 \\ 0 & x < 0\end{cases}\]
Symbols: \(\text{ReLU}(x) = \max(0,x)\); undefined at x=0 (use subgradient 0 or 1 in practice).
Intuition: Acts as a binary gate: passes gradient through if the neuron was active (x>0), blocks it otherwise. The dying ReLU problem: if always x≤0, neuron gets no gradient.

Q: What is the vector calculus identity for gradient of a dot product?
A: \[\nabla_{\mathbf{x}}(\mathbf{a}^\top\mathbf{x}) = \mathbf{a}, \quad \nabla_{\mathbf{x}}(\mathbf{x}^\top A\mathbf{x}) = 2A\mathbf{x} \text{ (A symmetric)}\]
Symbols: \(\mathbf{a}\) = constant vector; \(A\) = symmetric matrix.
Intuition: Linear term: gradient is the coefficient vector. Quadratic term: gradient is 2Ax. These two identities cover almost all gradient computations in linear algebra and regression.

Q: What is the matrix calculus identity for ∂tr(AX)/∂X?
A: \[\frac{\partial}{\partial X}\mathrm{tr}(AX) = A^\top\]
Symbols: \(X\) = matrix variable; \(A\) = constant matrix; \(\mathrm{tr}\) = trace.
Intuition: Trace of matrix product is a scalar, so its gradient w.r.t. X is a matrix. This identity is used in deriving MLE for Gaussian distributions and linear regression with matrix weights.

Q: What is the derivative of the matrix log-determinant?
A: \[\frac{\partial}{\partial A}\log\det(A) = A^{-\top} = (A^{-1})^\top\]
Symbols: \(\det(A)\) = determinant; \(A^{-\top}\) = inverse transpose.
Intuition: Appears in the log-likelihood of Gaussian distributions where the normalizing constant contains log det(Σ). For symmetric A: simplifies to A^{-1}.

Q: What is a smooth approximation to the max function?
A: \[\mathrm{softmax\_max}(\mathbf{z}) \approx \log\sum_i e^{z_i/T} \cdot T\]
Symbols: \(T\) = temperature; \(T \to 0\) → hard max; \(T \to \infty\) → average.
Intuition: The log-sum-exp is a smooth, differentiable upper bound on the maximum. Temperature T controls the smoothness. Widely used because it's differentiable everywhere, unlike argmax.

Q: What is the score function in statistics?
A: \[s(\theta) = \nabla_\theta \log p(x|\theta) = \frac{\nabla_\theta p(x|\theta)}{p(x|\theta)}\]
Symbols: \(s(\theta)\) = score; \(p(x|\theta)\) = likelihood; its expectation under p is zero.
Intuition: The gradient of the log-likelihood. Has zero mean under the model: E[s(θ)]=0. Its covariance is the Fisher information matrix — measures how much the distribution changes with θ.

Q: What is the Fisher information matrix?
A: \[\mathcal{I}(\theta) = \mathbb{E}\left[\nabla_\theta \log p(x|\theta) \, (\nabla_\theta \log p(x|\theta))^\top\right]\]
Symbols: \(\mathcal{I}(\theta)\) = Fisher information; outer product of score vectors.
Intuition: Measures how informative data is about θ. Used in natural gradient descent, the Cramér-Rao bound (variance lower bound for estimators), and second-order optimization.

Q: What is the natural gradient?
A: \[\tilde{\nabla}_\theta \mathcal{L} = \mathcal{I}(\theta)^{-1} \nabla_\theta \mathcal{L}\]
Symbols: \(\mathcal{I}(\theta)^{-1}\) = inverse Fisher information; \(\tilde{\nabla}\) = natural gradient.
Intuition: Adjusts the gradient direction to account for the geometry of the probability distribution (Riemannian geometry). Less sensitive to parameterization. Used in TRPO and natural policy gradient.

Q: What is the gradient of the Frobenius norm of a matrix?
A: \[\frac{\partial}{\partial W}\|W\|_F^2 = 2W\]
Symbols: \(\|W\|_F^2 = \mathrm{tr}(W^\top W) = \sum_{ij} W_{ij}^2\).
Intuition: The Frobenius norm is just the L2 norm of the flattened weight matrix. Its gradient is 2W — used directly in L2 weight decay regularization gradient.

Q: What are the Kolmogorov axioms of probability?
A: \[P(\Omega)=1,\quad P(A)\geq 0,\quad P(A\cup B)=P(A)+P(B) \text{ if } A\cap B=\emptyset\]
Symbols: \(\Omega\) = sample space; \(A, B\) = events.
Intuition: Three axioms: probabilities are non-negative, the certain event has probability 1, and probabilities of mutually exclusive events add.

Q: What is the complement rule in probability?
A: \[P(A^c) = 1 - P(A)\]
Symbols: \(A^c\) = complement of A (everything not in A).
Intuition: The probability of the opposite event = 1 minus the original probability. Useful when it's easier to compute P(not A) than P(A).

Q: What is the inclusion-exclusion principle?
A: \[P(A \cup B) = P(A) + P(B) - P(A \cap B)\]
Symbols: \(\cup\) = union (or); \(\cap\) = intersection (and).
Intuition: Adding P(A)+P(B) double-counts the overlap, so subtract P(A∩B). Generalizes to more events with alternating additions and subtractions.

Q: What is conditional probability?
A: \[P(A|B) = \frac{P(A \cap B)}{P(B)}, \quad P(B) > 0\]
Symbols: \(P(A|B)\) = probability of A given B occurred; \(P(A \cap B)\) = joint probability.
Intuition: Restricts the sample space to B and asks: within B, what fraction has A? Used everywhere in ML — likelihood p(x|θ), posterior p(θ|x).

Q: What does it mean for two events to be independent?
A: \[A \perp B \iff P(A \cap B) = P(A)P(B)\]
Symbols: \(\perp\) = independence; \(P(A|B) = P(A)\) (B tells us nothing about A).
Intuition: Knowing B happened doesn't change probability of A. Independence is a powerful simplifying assumption used in Naïve Bayes and factored distributions.

Q: What is the law of total probability?
A: \[P(A) = \sum_{i} P(A|B_i)P(B_i)\]
Symbols: \(\{B_i\}\) = partition of sample space (mutually exclusive, exhaustive).
Intuition: Marginalizes out an intermediate event B. Total probability is a weighted average of conditional probabilities. Used to derive marginals from joint distributions.

Q: What is Bayes' theorem?
A: \[P(\theta|x) = \frac{P(x|\theta)\,P(\theta)}{P(x)}\]
Symbols: \(P(\theta|x)\) = posterior; \(P(x|\theta)\) = likelihood; \(P(\theta)\) = prior; \(P(x)\) = evidence.
Intuition: Updates prior belief about θ after observing data x. Posterior ∝ likelihood × prior.

Q: What is the prior, likelihood, and posterior in Bayesian inference?
A: \[\underbrace{P(\theta|x)}_{\text{posterior}} \propto \underbrace{P(x|\theta)}_{\text{likelihood}} \cdot \underbrace{P(\theta)}_{\text{prior}}\]
Symbols: \(\propto\) = proportional to (ignoring the normalizing constant P(x)).
Intuition: Prior = what we believe before data. Likelihood = how probable the data is under θ. Posterior = updated belief combining both.

Q: What is the evidence (marginal likelihood) in Bayes' theorem?
A: \[P(x) = \int P(x|\theta)P(\theta)\,d\theta\]
Symbols: \(P(x)\) = normalizing constant ensuring posterior sums to 1.
Intuition: The marginal likelihood integrates out all possible θ. Often intractable for complex models — this is why variational inference and MCMC are needed.

Q: What is a Bayesian update for a single observation?
A: \[P(\theta|x_1,\ldots,x_n) \propto \left(\prod_{i=1}^n P(x_i|\theta)\right) P(\theta)\]
Symbols: i.i.d. observations; product of likelihoods × prior.
Intuition: With independent observations, likelihoods multiply. In log space: log posterior = Σ log likelihood + log prior. MAP finds the peak of this.

Q: What is a probability mass function (PMF)?
A: \[p(x) = P(X = x), \quad \sum_x p(x) = 1\]
Symbols: \(X\) = discrete random variable; \(p(x) \geq 0\) for all x.
Intuition: Assigns probability to each possible outcome of a discrete variable. Must be non-negative and sum to 1.

Q: What is a probability density function (PDF)?
A: \[f(x) \geq 0, \quad \int_{-\infty}^{\infty} f(x)\,dx = 1\]
Symbols: \(f(x)\) = density at x; \(P(a \leq X \leq b) = \int_a^b f(x)dx\).
Intuition: For continuous variables, probability at a single point is zero — instead, area under the curve over an interval gives probability.

Q: What is a cumulative distribution function (CDF)?
A: \[F(x) = P(X \leq x) = \int_{-\infty}^x f(t)\,dt\]
Symbols: \(F\) = CDF; monotone non-decreasing; \(F(-\infty)=0\), \(F(+\infty)=1\).
Intuition: Gives the cumulative probability up to x. Continuous, right-continuous, and non-decreasing. The PDF is the derivative of the CDF.

Q: What is the expected value of a discrete random variable?
A: \[\mathbb{E}[X] = \sum_x x \cdot p(x)\]
Symbols: \(p(x)\) = PMF; sum over all possible values.
Intuition: The probability-weighted average of all possible values. The 'center of mass' of the distribution. For continuous: \(\mathbb{E}[X] = \int x f(x)dx\).

Q: What is the linearity of expectation?
A: \[\mathbb{E}[aX + bY] = a\mathbb{E}[X] + b\mathbb{E}[Y]\]
Symbols: \(a, b\) = constants; holds even if X and Y are not independent.
Intuition: One of the most useful tools in probability — expectation distributes over sums and scales with constants, regardless of dependence structure.

Q: What is the law of the unconscious statistician (LOTUS)?
A: \[\mathbb{E}[g(X)] = \sum_x g(x)p(x)\]
Symbols: \(g\) = any function of X; no need to find distribution of g(X) first.
Intuition: Computes the expectation of a function of X directly from the distribution of X. E.g., E[X²] = Σ x² p(x). Used everywhere in variance and moment calculations.

Q: What is the variance of a random variable?
A: \[\mathrm{Var}(X) = \mathbb{E}[(X-\mu)^2] = \mathbb{E}[X^2] - (\mathbb{E}[X])^2\]
Symbols: \(\mu = \mathbb{E}[X]\); the second form is often easier to compute.
Intuition: Measures spread around the mean. Zero variance = constant variable. The shortcut E[X²]-(E[X])² avoids computing the mean first.

Q: How does variance behave under scaling and shifting?
A: \[\mathrm{Var}(aX + b) = a^2 \mathrm{Var}(X)\]
Symbols: \(a\) = scale factor; \(b\) = shift (additive constant has no effect on variance).
Intuition: Shifting doesn't change spread. Scaling by a squares in variance (because variance is about squared deviations).

Q: What is the standard deviation?
A: \[\sigma = \sqrt{\mathrm{Var}(X)}\]
Symbols: \(\sigma\) = standard deviation; same units as X unlike variance.
Intuition: The 'typical' deviation from the mean. More interpretable than variance because it's in the same units as X. Normal distribution is parameterized by μ and σ.

Q: What is the covariance between two random variables?
A: \[\mathrm{Cov}(X,Y) = \mathbb{E}[(X-\mu_X)(Y-\mu_Y)] = \mathbb{E}[XY] - \mathbb{E}[X]\mathbb{E}[Y]\]
Symbols: \(\mu_X, \mu_Y\) = means; \(\mathrm{Cov}(X,X) = \mathrm{Var}(X)\).
Intuition: Positive: X and Y tend to vary together. Negative: they vary oppositely. Zero: uncorrelated (but not necessarily independent).

Q: What is the Pearson correlation coefficient?
A: \[\rho(X,Y) = \frac{\mathrm{Cov}(X,Y)}{\sigma_X \sigma_Y} \in [-1,1]\]
Symbols: \(\sigma_X, \sigma_Y\) = standard deviations; \(|\rho|=1\) = perfect linear dependence.
Intuition: Normalizes covariance to [-1,1]. 1 = perfect positive linear correlation; -1 = perfect negative; 0 = uncorrelated. Only measures linear dependence.

Q: What is the Bernoulli distribution?
A: \[P(X=1) = p, \quad P(X=0) = 1-p, \quad \mathbb{E}[X] = p, \quad \mathrm{Var}(X) = p(1-p)\]
Symbols: \(p \in [0,1]\) = success probability; single binary trial.
Intuition: Simplest distribution — a single coin flip. Foundation for binary classification output. Maximum variance at p=0.5 (most uncertainty).

Q: What is the Binomial distribution?
A: \[P(X=k) = \binom{n}{k}p^k(1-p)^{n-k},\quad \mathbb{E}[X]=np,\quad \mathrm{Var}(X)=np(1-p)\]
Symbols: \(n\) = number of trials; \(k\) = number of successes; \(\binom{n}{k}\) = binomial coefficient.
Intuition: Sum of n independent Bernoulli(p) trials. Approaches Normal(np, np(1-p)) for large n by CLT.

Q: What is the Poisson distribution?
A: \[P(X=k) = \frac{\lambda^k e^{-\lambda}}{k!}, \quad k \in \{0,1,2,\ldots\}, \quad \mathbb{E}[X] = \mathrm{Var}(X) = \lambda\]
Symbols: \(\lambda > 0\) = average rate; \(k!\) = factorial.
Intuition: Models count of rare events in a fixed interval (arrivals, word counts). Unique property: mean equals variance. Widely used in NLP language models.

Q: What is the Geometric distribution?
A: \[P(X=k) = (1-p)^{k-1}p, \quad k=1,2,\ldots, \quad \mathbb{E}[X]=\frac{1}{p}\]
Symbols: \(p\) = success probability; \(k\) = trial on which first success occurs.
Intuition: Number of trials until first success. Memoryless property: P(X>m+n|X>m) = P(X>n). Only discrete memoryless distribution.

Q: What is the Gaussian (Normal) distribution?
A: \[\mathcal{N}(x|\mu,\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)\]
Symbols: \(\mu\) = mean; \(\sigma^2\) = variance; \(\sigma\) = standard deviation; \(x \in (-\infty,\infty)\).
Intuition: The bell curve. Most important distribution: arises from CLT, maximum entropy for fixed mean/variance, closed under linear operations. Cornerstone of ML.

Q: What is the standard normal distribution?
A: \[Z = \frac{X - \mu}{\sigma} \sim \mathcal{N}(0,1)\]
Symbols: \(Z\) = standardized variable; \(\mu\) = mean; \(\sigma\) = std.
Intuition: Shifts and scales any Normal to have mean 0, variance 1. Standardization allows comparing values across different scales — the z-score measures how many standard deviations from the mean.

Q: What is the 68-95-99.7 rule for the Normal distribution?
A: \[P(\mu - k\sigma \leq X \leq \mu + k\sigma) \approx \begin{cases}68.3\% & k=1 \\ 95.4\% & k=2 \\ 99.7\% & k=3\end{cases}\]
Symbols: \(k\) = number of standard deviations from mean; \(\mu\) = mean; \(\sigma\) = std.
Intuition: Almost all mass of a Gaussian is within 3σ of the mean. Used to set thresholds, detect outliers, and reason about prediction intervals.

Q: What is the multivariate Gaussian distribution?
A: \[\mathcal{N}(\mathbf{x}|\boldsymbol{\mu},\Sigma) = \frac{1}{(2\pi)^{d/2}|\Sigma|^{1/2}}\exp\!\left(-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu})^\top\Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu})\right)\]
Symbols: \(\boldsymbol{\mu} \in \mathbb{R}^d\) = mean vector; \(\Sigma\) = covariance matrix (PSD); \(|\Sigma|\) = determinant.
Intuition: d-dimensional generalization of the Gaussian. The exponent is the squared Mahalanobis distance. Elliptical contours aligned with eigenvectors of Σ.

Q: What is the Mahalanobis distance?
A: \[d_M(\mathbf{x},\boldsymbol{\mu}) = \sqrt{(\mathbf{x}-\boldsymbol{\mu})^\top \Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu})}\]
Symbols: \(\Sigma^{-1}\) = precision matrix; accounts for correlation and scale.
Intuition: Euclidean distance normalized by the covariance structure. Scale-invariant and accounts for correlated features. Measures how many standard deviations x is from μ in any direction.

Q: What is the Categorical distribution?
A: \[P(X=k) = p_k, \quad \sum_{k=1}^K p_k = 1\]
Symbols: \(K\) = number of categories; \(p_k \geq 0\) = probability of category k.
Intuition: Generalization of Bernoulli to K outcomes. Parameterized by a probability vector. Softmax output represents a Categorical distribution over class labels.

Q: What is the Dirichlet distribution?
A: \[\mathrm{Dir}(\boldsymbol{\alpha}) \propto \prod_{k=1}^K p_k^{\alpha_k-1}, \quad \sum_k p_k = 1\]
Symbols: \(\boldsymbol{\alpha}\) = concentration parameters; \(p_k\) = probability values.
Intuition: Distribution over probability vectors (simplex). Conjugate prior for the Categorical/Multinomial. Large α concentrates mass near the mean; small α spreads mass to corners.

Q: What is the Beta distribution?
A: \[\mathrm{Beta}(p|\alpha,\beta) \propto p^{\alpha-1}(1-p)^{\beta-1}, \quad p \in [0,1]\]
Symbols: \(\alpha, \beta > 0\) = shape parameters; \(\mathbb{E}[p] = \alpha/(\alpha+\beta)\).
Intuition: Distribution over probabilities in [0,1]. Conjugate prior for Bernoulli/Binomial. α=β=1 is uniform; large α,β concentrates near α/(α+β).

Q: What is the joint probability distribution?
A: \[p(x, y) = P(X=x, Y=y)\]
Symbols: \(p(x,y)\) = probability of both X=x and Y=y occurring.
Intuition: Describes the combined behavior of two (or more) variables simultaneously. All marginals and conditionals can be derived from the joint.

Q: What is the marginal distribution?
A: \[p(x) = \sum_y p(x,y) \quad \text{or} \quad p(x) = \int p(x,y)\,dy\]
Symbols: Sum/integral over all values of Y (marginalize out Y).
Intuition: 'Sum out' the variable you don't care about to get the distribution of a single variable. Marginal = projection of the joint onto one axis.

Q: How does the conditional distribution relate to the joint?
A: \[p(x|y) = \frac{p(x,y)}{p(y)}\]
Symbols: \(p(y) = \sum_x p(x,y)\) = marginal of Y.
Intuition: Slice through the joint at a fixed y, then normalize. The conditional distribution is the joint normalized to integrate to 1 over x.

Q: What is Maximum Likelihood Estimation (MLE)?
A: \[\hat{\theta}_{\mathrm{MLE}} = \arg\max_\theta \prod_{i=1}^N p(x_i|\theta) = \arg\max_\theta \sum_{i=1}^N \log p(x_i|\theta)\]
Symbols: \(\hat{\theta}\) = MLE estimate; log converts product to sum for numerical stability.
Intuition: Find parameters that make the observed data most probable. Equivalent to minimizing negative log-likelihood — standard training objective in most ML models.

Q: What is the MLE for the mean of a Gaussian?
A: \[\hat{\mu}_{\mathrm{MLE}} = \frac{1}{N}\sum_{i=1}^N x_i = \bar{x}\]
Symbols: \(N\) = number of samples; \(\bar{x}\) = sample mean.
Intuition: The sample mean maximizes the Gaussian likelihood. Derive by setting ∂ log L/∂μ = 0. The MLE for σ² is the biased sample variance (divides by N not N-1).

Q: What is the MLE for the variance of a Gaussian?
A: \[\hat{\sigma}^2_{\mathrm{MLE}} = \frac{1}{N}\sum_{i=1}^N (x_i - \hat{\mu})^2\]
Symbols: Note: divides by N (biased); unbiased estimator uses N-1.
Intuition: The MLE underestimates variance (biased). The corrected unbiased estimator (Bessel's correction) divides by N-1 instead.

Q: Why do we maximize log-likelihood instead of likelihood?
A: \[\log \prod_{i=1}^N p(x_i|\theta) = \sum_{i=1}^N \log p(x_i|\theta)\]
Symbols: Log converts product to sum; log is monotone so maximizing log L ↔ maximizing L.
Intuition: Products of small probabilities cause numerical underflow. Log converts to sums, avoids underflow, and log is monotone so the argmax doesn't change.

Q: What is Maximum A Posteriori (MAP) estimation?
A: \[\hat{\theta}_{\mathrm{MAP}} = \arg\max_\theta \log p(\theta|X) = \arg\max_\theta \left[\sum_i \log p(x_i|\theta) + \log p(\theta)\right]\]
Symbols: \(\log p(\theta)\) = log prior; adds regularization effect.
Intuition: Finds the mode of the posterior. MLE ignores the prior; MAP incorporates prior beliefs. L2 regularization = Gaussian prior; L1 regularization = Laplace prior.

Q: How does MAP relate to regularized MLE?
A: \[\hat{\theta}_{\mathrm{MAP}} = \arg\min_\theta \left[-\sum_i \log p(x_i|\theta) + \lambda\|\theta\|^2\right]\]
Symbols: \(\lambda\|\theta\|^2\) = L2 regularization term; \(\lambda\) = regularization strength.
Intuition: A Gaussian prior \(p(\theta) \propto e^{-\lambda\|\theta\|^2}\) corresponds to L2 regularization. MAP = MLE + regularizer — Bayesian interpretation of weight decay.

Q: What is a conjugate prior?
A: \[p(\theta|x) \propto p(x|\theta)p(\theta) \implies p(\theta|x) \text{ same family as } p(\theta)\]
Symbols: Family closed under Bayesian updating.
Intuition: The posterior has the same distributional form as the prior — making the update analytic. Examples: Beta is conjugate to Bernoulli; Dirichlet to Categorical; Gaussian to Gaussian.

Q: What is the Beta-Bernoulli conjugate model?
A: \[\theta \sim \mathrm{Beta}(\alpha,\beta), \quad X|\theta \sim \mathrm{Bern}(\theta) \implies \theta|X \sim \mathrm{Beta}(\alpha+\#\text{heads}, \beta+\#\text{tails})\]
Symbols: \(\alpha, \beta\) = prior pseudo-counts; updated by adding observed counts.
Intuition: Each observation just increments a counter. Prior α,β act like imaginary previous observations. More data → posterior concentrates around true probability.

Q: What is the Law of Large Numbers (LLN)?
A: \[\bar{X}_N = \frac{1}{N}\sum_{i=1}^N X_i \xrightarrow{N\to\infty} \mathbb{E}[X]\]
Symbols: \(\bar{X}_N\) = sample mean; converges in probability to the true mean.
Intuition: Sample averages converge to the population mean as sample size grows. Justifies Monte Carlo: estimate expectations by averaging samples.

Q: What is the Central Limit Theorem (CLT)?
A: \[\sqrt{N}\left(\bar{X}_N - \mu\right) \xrightarrow{d} \mathcal{N}(0, \sigma^2)\]
Symbols: \(\mu = \mathbb{E}[X]\); \(\sigma^2 = \mathrm{Var}(X)\); \(\xrightarrow{d}\) = convergence in distribution.
Intuition: The sample mean of many i.i.d. variables, regardless of their distribution, converges to a Gaussian. Justifies assuming normality in large samples.

Q: What is the practical form of the CLT for the sample mean?
A: \[\bar{X}_N \approx \mathcal{N}\!\left(\mu, \frac{\sigma^2}{N}\right)\]
Symbols: \(\sigma^2/N\) = variance of sample mean decreases with N.
Intuition: The sample mean is approximately Gaussian, with variance shrinking as 1/N. Basis for confidence intervals and hypothesis testing.

Q: What is the bias-variance decomposition of MSE?
A: \[\mathbb{E}[(\hat{f}(x) - f(x))^2] = \mathrm{Bias}^2 + \mathrm{Variance} + \sigma^2\]
Symbols: \(\mathrm{Bias} = \mathbb{E}[\hat{f}(x)] - f(x)\); \(\mathrm{Variance} = \mathbb{E}[(\hat{f}(x)-\mathbb{E}[\hat{f}(x)])^2]\); \(\sigma^2\) = irreducible noise.
Intuition: Prediction error decomposes into: systematic error (bias), sensitivity to training data (variance), and irreducible noise. The classic tradeoff.

Q: What is the bias-variance tradeoff?
A: \[\text{Bias}^2 \downarrow \implies \text{Variance} \uparrow \text{ (typically)}\]
Symbols: Complex models have low bias but high variance; simple models have high bias but low variance.
Intuition: Increasing model complexity reduces underfitting (bias) but increases overfitting (variance). Regularization, ensembling, and more data shift this tradeoff.

Q: What is the bias of an estimator?
A: \[\mathrm{Bias}(\hat{\theta}) = \mathbb{E}[\hat{\theta}] - \theta\]
Symbols: \(\hat{\theta}\) = estimator; \(\theta\) = true parameter; unbiased iff \(\mathbb{E}[\hat{\theta}] = \theta\).
Intuition: An estimator is unbiased if it's correct on average. The MLE variance estimator (÷N) is biased; the corrected estimator (÷N-1) is unbiased.

Q: What is the moment generating function (MGF)?
A: \[M_X(t) = \mathbb{E}[e^{tX}]\]
Symbols: \(t \in \mathbb{R}\); k-th moment: \(\mathbb{E}[X^k] = M_X^{(k)}(0)\) (k-th derivative at 0).
Intuition: A function that encodes all moments. Differentiate k times and evaluate at t=0 to get E[X^k]. MGF uniquely characterizes a distribution (when it exists).

Q: What is Monte Carlo estimation?
A: \[\mathbb{E}_p[f(x)] \approx \frac{1}{N}\sum_{i=1}^N f(x_i), \quad x_i \sim p\]
Symbols: \(N\) = number of samples; converges by LLN.
Intuition: Estimate intractable expectations by averaging function values over random samples. Error ∝ 1/√N (by CLT). Foundation of approximate inference and RL policy gradient.

Q: What is importance sampling?
A: \[\mathbb{E}_p[f(x)] = \mathbb{E}_q\!\left[\frac{p(x)}{q(x)}f(x)\right] \approx \frac{1}{N}\sum_{i=1}^N \frac{p(x_i)}{q(x_i)}f(x_i),\quad x_i \sim q\]
Symbols: \(q\) = proposal distribution; \(p(x)/q(x)\) = importance weight.
Intuition: Sample from a different (more tractable) distribution q and correct via weights. Efficient when p is hard to sample from but q is close to p.

Q: What is conditional independence?
A: \[X \perp Y | Z \iff P(X,Y|Z) = P(X|Z)P(Y|Z)\]
Symbols: \(X \perp Y | Z\) = X is independent of Y given Z.
Intuition: Once we know Z, X gives no extra information about Y. Fundamental in graphical models (Bayes nets) — defines the conditional independence structure of the graph.

Q: What is the KL divergence in the context of Bayes' theorem?
A: \[D_{\mathrm{KL}}(q\|p) = \mathbb{E}_q\!\left[\log\frac{q(\theta)}{p(\theta|x)}\right]\]
Symbols: \(q(\theta)\) = variational distribution; \(p(\theta|x)\) = true posterior.
Intuition: Variational inference minimizes KL divergence between a tractable q and the intractable posterior. Minimizing KL is equivalent to maximizing the ELBO.

Q: What is the exponential family of distributions?
A: \[p(x|\eta) = h(x)\exp(\eta^\top T(x) - A(\eta))\]
Symbols: \(\eta\) = natural parameter; \(T(x)\) = sufficient statistic; \(A(\eta)\) = log-partition function.
Intuition: A broad family including Gaussian, Bernoulli, Poisson, Gamma. Has conjugate priors, sufficient statistics, and convenient optimization properties.

Q: What is a sufficient statistic?
A: \[T(X) \text{ is sufficient for } \theta \iff p(X|T(X),\theta) = p(X|T(X))\]
Symbols: \(T(X)\) = statistic (function of data); sufficient means it captures all θ-relevant information.
Intuition: You don't lose information about θ by compressing X to T(X). For Gaussian: sample mean and variance are sufficient — raw data adds nothing for estimating μ,σ.

Q: What is the Cramér-Rao lower bound?
A: \[\mathrm{Var}(\hat{\theta}) \geq \frac{1}{\mathcal{I}(\theta)}\]
Symbols: \(\mathcal{I}(\theta)\) = Fisher information; \(\hat{\theta}\) = any unbiased estimator.
Intuition: No unbiased estimator can have lower variance than the inverse Fisher information. Defines the best achievable precision — efficient estimators (like MLE under regularity) achieve this bound.

Q: What is the log-normal distribution?
A: \[\log X \sim \mathcal{N}(\mu,\sigma^2) \implies \mathbb{E}[X] = e^{\mu + \sigma^2/2}\]
Symbols: \(X > 0\); a positive-valued variable whose log is Gaussian.
Intuition: Models multiplicative processes (growth rates, prices, word frequencies). Heavy right tail — mean > median. Common in language (Zipf's law approximation).

Q: What is the Laplace distribution?
A: \[p(x|\mu,b) = \frac{1}{2b}\exp\!\left(-\frac{|x-\mu|}{b}\right)\]
Symbols: \(\mu\) = location; \(b\) = scale; heavier tails than Gaussian.
Intuition: Symmetric distribution with sharper peak and heavier tails than Gaussian. MAP estimation with Laplace prior yields L1 regularization (promotes sparsity).

Q: What is the Student's t-distribution?
A: \[p(t|\nu) \propto \left(1 + \frac{t^2}{\nu}\right)^{-(\nu+1)/2}\]
Symbols: \(\nu\) = degrees of freedom; heavier tails than Gaussian; \(\nu \to \infty\) → Gaussian.
Intuition: Robust to outliers due to heavy tails. Used when sample size is small and σ is unknown. Basis for t-tests and robust regression.

Q: What is the Gamma distribution?
A: \[p(x|k,\theta) = \frac{x^{k-1}e^{-x/\theta}}{\theta^k \Gamma(k)}, \quad x>0\]
Symbols: \(k\) = shape; \(\theta\) = scale; \(\Gamma(k)\) = gamma function; \(\mathbb{E}[X]=k\theta\).
Intuition: Models positive-valued quantities (waiting times, variances). The Gamma is conjugate to Gaussian precision. Chi-squared is a special case.

Q: What is the negative log-likelihood as a loss function?
A: \[\mathcal{L}(\theta) = -\sum_{i=1}^N \log p(x_i|\theta)\]
Symbols: \(p(x_i|\theta)\) = model probability of i-th example; minimizing NLL = maximizing likelihood.
Intuition: Standard training objective for probabilistic models. For a Gaussian output with fixed variance: NLL reduces to MSE. For Bernoulli output: NLL reduces to binary cross-entropy.

Q: What is the variance of a sum of random variables?
A: \[\mathrm{Var}\!\left(\sum_i X_i\right) = \sum_i \mathrm{Var}(X_i) + 2\sum_{i<j}\mathrm{Cov}(X_i,X_j)\]
Symbols: \(\mathrm{Cov}\) = covariance; for independent variables: \(\mathrm{Var}(\sum X_i)=\sum\mathrm{Var}(X_i)\).
Intuition: Variances add only for independent variables. Positively correlated variables increase total variance (less diversity); negatively correlated decrease it.

Q: What is the Gaussian PDF for a diagonal covariance?
A: \[\mathcal{N}(\mathbf{x}|\boldsymbol{\mu}, \mathrm{diag}(\boldsymbol{\sigma}^2)) = \prod_{i=1}^d \mathcal{N}(x_i|\mu_i,\sigma_i^2)\]
Symbols: \(\boldsymbol{\sigma}^2\) = vector of per-dimension variances; factorizes into independent 1D Gaussians.
Intuition: Diagonal covariance = independent features. The joint density factorizes into a product. This is the assumption in many variational autoencoders and Gaussian Naïve Bayes.

Q: What is the entropy of a Gaussian distribution?
A: \[H(\mathcal{N}(\mu,\sigma^2)) = \frac{1}{2}\ln(2\pi e \sigma^2)\]
Symbols: \(e\) = Euler's number; entropy in nats (using ln).
Intuition: Entropy increases with σ — wider spread = more uncertainty. Among all distributions with fixed variance, the Gaussian has maximum entropy. Key in information-theoretic analyses.

Q: What is the maximum entropy principle?
A: \[p^* = \arg\max_p H(p) \text{ subject to constraints on } \mathbb{E}[f(x)]\]
Symbols: \(H(p)\) = entropy; constraints fix moments (e.g., mean and variance).
Intuition: The maximum entropy distribution is the 'least committed' distribution satisfying known constraints. With mean and variance constraints → Gaussian. With no constraints → uniform distribution.

Q: What is the Markov property?
A: \[P(X_{t+1}|X_t, X_{t-1},\ldots,X_1) = P(X_{t+1}|X_t)\]
Symbols: \(X_t\) = state at time t; future depends only on present state.
Intuition: The future is independent of the past given the present. Simplifies sequential models enormously. Foundation for Markov chains, HMMs, MDPs, and sequence modeling.

Q: What is an i.i.d. assumption?
A: \[x_1,\ldots,x_N \sim p(x) \text{ i.i.d.} \implies p(x_1,\ldots,x_N) = \prod_{i=1}^N p(x_i)\]
Symbols: i.i.d. = independently and identically distributed; joint factorizes.
Intuition: Each sample is drawn independently from the same distribution. Standard assumption in supervised learning — training examples are exchangeable. Violated in time series.

Q: What is the predictive distribution in Bayesian inference?
A: \[p(x_{\text{new}}|x_{1:N}) = \int p(x_{\text{new}}|\theta)p(\theta|x_{1:N})\,d\theta\]
Symbols: \(p(\theta|x_{1:N})\) = posterior; integrates over all possible θ.
Intuition: Instead of plugging in a single θ, average predictions over all plausible models weighted by their posterior probability. More robust to overfitting than MAP/MLE predictions.

Q: What is the evidence lower bound (ELBO)?
A: \[\log p(x) \geq \mathbb{E}_q[\log p(x,z)] - \mathbb{E}_q[\log q(z)] = \mathcal{L}\]
Symbols: \(z\) = latent variable; \(q(z)\) = variational distribution; \(\mathcal{L}\) = ELBO.
Intuition: A lower bound on the log-evidence. Maximizing ELBO simultaneously maximizes likelihood and minimizes KL(q||p(z|x)). Foundation of variational autoencoders (VAEs).

Q: What is the reparameterization gradient estimator?
A: \[\nabla_\phi \mathbb{E}_{q_\phi}[f(z)] = \mathbb{E}_\epsilon[\nabla_\phi f(g(\epsilon,\phi))],\quad z = g(\epsilon,\phi),\;\epsilon \sim p(\epsilon)\]
Symbols: \(\phi\) = variational parameters; \(g\) = differentiable transform; \(\epsilon\) = noise.
Intuition: Moves the gradient inside the expectation by reparameterizing z as a deterministic function of noise ε and parameters φ. Low-variance gradient estimator used in VAE training.

Q: What is the Chebyshev inequality?
A: \[P(|X - \mu| \geq k\sigma) \leq \frac{1}{k^2}\]
Symbols: \(\mu = \mathbb{E}[X]\); \(\sigma^2 = \mathrm{Var}(X)\); \(k > 0\).
Intuition: At least 1-1/k² of the probability mass lies within k standard deviations of the mean, for any distribution with finite variance. Weaker than Gaussian but universally applicable.

Q: What is a p-value in hypothesis testing?
A: \[p\text{-value} = P(\text{test statistic} \geq t_{\text{obs}} | H_0)\]
Symbols: \(H_0\) = null hypothesis; \(t_{\text{obs}}\) = observed test statistic.
Intuition: Probability of observing results at least as extreme as yours, assuming H0 is true. Small p-value → data unlikely under H0 → evidence against H0. Not the probability H0 is true.

Q: What is the Gaussian kernel (RBF kernel) in machine learning?
A: \[k(x, x') = \exp\!\left(-\frac{\|x-x'\|^2}{2\ell^2}\right)\]
Symbols: \(\ell\) = length-scale; measures decay of similarity with distance.
Intuition: Measures similarity between points: 1 when identical, decays to 0 as distance increases. Widely used in SVMs and Gaussian processes. Length-scale ℓ controls how fast similarity decays.

Q: What is the law of iterated expectations?
A: \[\mathbb{E}[X] = \mathbb{E}_Y[\mathbb{E}[X|Y]]\]
Symbols: \(\mathbb{E}[X|Y]\) = conditional expectation; outer expectation over Y.
Intuition: Average the conditional averages to get the unconditional average. Used in deriving bias-variance tradeoff, EM algorithm, and regression analysis.

Q: What is the law of total variance?
A: \[\mathrm{Var}(X) = \mathbb{E}[\mathrm{Var}(X|Y)] + \mathrm{Var}(\mathbb{E}[X|Y])\]
Symbols: First term = average within-group variance; second = between-group variance.
Intuition: Total variance splits into variance explained by Y (the between term) and unexplained variance (within term). Used in Rao-Blackwell theorem and random effects models.

Q: What is the moment matching condition for distribution fitting?
A: \[\mathbb{E}_p[x^k] = \mathbb{E}_q[x^k] \text{ for all } k\]
Symbols: \(p\) = true distribution; \(q\) = fitted distribution; match moments up to order K.
Intuition: Fit a distribution by matching its moments (mean, variance, skew, ...) to the data. Method of moments estimation. Simpler than MLE but sometimes less efficient.

Q: What is the Gaussian mixture model (GMM)?
A: \[p(x) = \sum_{k=1}^K \pi_k \mathcal{N}(x|\mu_k, \Sigma_k), \quad \sum_k \pi_k = 1\]
Symbols: \(\pi_k\) = mixture weights; \(K\) = number of components; \(\mu_k, \Sigma_k\) = component parameters.
Intuition: A weighted sum of Gaussians. Universal approximator of smooth densities. Fit by EM algorithm: E-step computes soft cluster assignments, M-step updates parameters.

Q: What is the EM algorithm?
A: \[\theta^{(t+1)} = \arg\max_\theta \mathbb{E}_{q^{(t)}(z)}[\log p(x,z|\theta)]\]
Symbols: \(z\) = latent variables; \(q^{(t)}(z) = p(z|x,\theta^{(t)})\) = posterior at step t.
Intuition: Alternates between E-step (compute expected sufficient statistics given current θ) and M-step (maximize expected complete-data log-likelihood). Monotonically increases the marginal likelihood.

Q: What is the exponential distribution?
A: \[p(x|\lambda) = \lambda e^{-\lambda x}, \quad x \geq 0, \quad \mathbb{E}[X] = 1/\lambda\]
Symbols: \(\lambda > 0\) = rate parameter; special case of Gamma with shape k=1.
Intuition: Models time until first event in a Poisson process. Memoryless property. Used in survival analysis, network queuing, and modeling inter-arrival times.

Q: What is a sufficient condition for unbiasedness of an estimator?
A: \[\mathbb{E}_\theta[\hat{\theta}(X)] = \theta \text{ for all } \theta\]
Symbols: \(\hat{\theta}\) = estimator function of data X; equality holds for all possible true values.
Intuition: An unbiased estimator's sampling distribution is centered at the true parameter. The sample mean is unbiased for μ; MLE variance estimate is biased (use Bessel's correction).

Q: What is bootstrapping in statistics?
A: \[\hat{F}(t) = \frac{1}{B}\sum_{b=1}^B \mathbf{1}[\hat{\theta}^{(b)} \leq t]\]
Symbols: \(B\) = bootstrap resamples; \(\hat{\theta}^{(b)}\) = estimate on b-th resample; \(\hat{F}\) = empirical distribution of statistic.
Intuition: Resample the data with replacement many times to estimate the sampling distribution of a statistic without analytical derivation. Basis for bootstrap confidence intervals.

Q: What is the connection between MLE and cross-entropy minimization?
A: \[\hat{\theta}_{\mathrm{MLE}} = \arg\min_\theta H(p_{\mathrm{data}}, p_\theta) = \arg\min_\theta D_{\mathrm{KL}}(p_{\mathrm{data}}\|p_\theta)\]
Symbols: \(H(p_{\mathrm{data}}, p_\theta)\) = cross-entropy; \(p_{\mathrm{data}}\) = empirical data distribution.
Intuition: Maximizing log-likelihood ≡ minimizing cross-entropy ≡ minimizing KL divergence from data distribution to model. Unifies MLE with information-theoretic objectives.

Q: What is a confidence interval?
A: \[P(\hat{\theta}_L \leq \theta \leq \hat{\theta}_U) = 1 - \alpha\]
Symbols: \(1-\alpha\) = confidence level (e.g., 95%); \([\hat{\theta}_L, \hat{\theta}_U]\) = interval bounds depending on data.
Intuition: If you repeat the experiment many times, 95% of such intervals would contain the true θ. It's about the procedure, not the probability that θ lies in any one specific interval.

Q: What is the self-information (surprisal) of an event?
A: \[I(x) = -\log_2 p(x)\]
Symbols: \(p(x)\) = probability of event x; log base 2 gives units in bits.
Intuition: Rare events are more surprising and carry more information. P=1 → 0 bits (no surprise). P=0.5 → 1 bit. P=0.25 → 2 bits. Doubling rarity adds 1 bit.

Q: Why is the log function used to define information?
A: \[I(x_1 \cap x_2) = I(x_1) + I(x_2) \text{ if } x_1 \perp x_2\]
Symbols: \(\log p(x_1 x_2) = \log p(x_1) + \log p(x_2)\) for independent events.
Intuition: We want information to be additive for independent events. Log of a product = sum of logs, which achieves this. A natural, axiomatically justified choice.

Q: What are the units of information depending on the log base?
A: \[I(x) = \begin{cases}-\log_2 p(x) & \text{bits (shannons)} \\ -\ln p(x) & \text{nats} \\ -\log_{10} p(x) & \text{hartleys}\end{cases}\]
Symbols: Different log bases give different units but same mathematical structure.
Intuition: In ML, nats (natural log) are most common in theory; bits (log base 2) in communication theory. Conversion: 1 nat = log₂(e) ≈ 1.44 bits.

Q: What is Shannon entropy?
A: \[H(X) = -\sum_{x} p(x)\log_2 p(x)\]
Symbols: \(H(X)\) = entropy in bits; \(p(x)\) = PMF; by convention \(0\log 0 = 0\).
Intuition: Expected self-information — average surprise. Maximum when all outcomes are equally likely (maximum uncertainty). Zero when outcome is certain.

Q: What is the maximum entropy for a K-outcome discrete variable?
A: \[H(X) \leq \log_2 K, \text{ achieved when } p(x) = 1/K \text{ for all } x\]
Symbols: \(K\) = number of outcomes; uniform distribution maximizes entropy.
Intuition: Entropy is maximized by the uniform distribution. Any bias toward some outcomes reduces uncertainty. A fair die has entropy log₂(6) ≈ 2.58 bits.

Q: What is the entropy of a Bernoulli variable?
A: \[H(p) = -p\log_2 p - (1-p)\log_2(1-p)\]
Symbols: \(p\) = probability of success; \(H(0)=H(1)=0\); \(H(0.5)=1\) bit.
Intuition: Binary entropy function. Max at p=0.5 (1 bit — perfectly uncertain). Zero when outcome is determined (p=0 or p=1). Used in decision tree splitting criteria.

Q: What is the differential entropy for a continuous distribution?
A: \[h(X) = -\int p(x)\ln p(x)\,dx\]
Symbols: \(h(X)\) = differential entropy (can be negative!); integral instead of sum.
Intuition: Continuous analog of Shannon entropy. Unlike discrete entropy, it can be negative and is not invariant to reparameterization. For Gaussian: h = ½ln(2πeσ²) nats.

Q: What is joint entropy?
A: \[H(X,Y) = -\sum_{x,y} p(x,y)\log p(x,y)\]
Symbols: \(p(x,y)\) = joint PMF; \(H(X,Y) \leq H(X)+H(Y)\) with equality iff X⊥Y.
Intuition: Total uncertainty in both X and Y together. If independent, they add; if dependent, knowing one reduces uncertainty of the other, so joint < sum.

Q: What is conditional entropy H(X|Y)?
A: \[H(X|Y) = -\sum_{x,y} p(x,y)\log p(x|y) = H(X,Y) - H(Y)\]
Symbols: \(H(X|Y)\) = remaining uncertainty in X after observing Y; always \(\leq H(X)\).
Intuition: Average uncertainty in X given we know Y. Conditioning reduces entropy (or leaves it unchanged if X⊥Y). Chain rule: H(X,Y) = H(Y) + H(X|Y).

Q: What is the chain rule of entropy?
A: \[H(X_1,\ldots,X_n) = \sum_{i=1}^n H(X_i|X_1,\ldots,X_{i-1})\]
Symbols: Decomposes joint entropy into sum of conditional entropies.
Intuition: Total uncertainty equals: uncertainty in X₁, plus additional uncertainty in X₂ given X₁, plus additional uncertainty in X₃ given X₁,X₂, etc. Used in language model perplexity.

Q: What is mutual information?
A: \[I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X) = H(X)+H(Y)-H(X,Y)\]
Symbols: \(I(X;Y) \geq 0\); equals 0 iff X⊥Y.
Intuition: The reduction in uncertainty about X due to knowing Y (and vice versa). Measures statistical dependence. Symmetric: knowing X reduces Y's uncertainty by the same amount.

Q: How is mutual information expressed as a KL divergence?
A: \[I(X;Y) = D_{\mathrm{KL}}(p(x,y)\|p(x)p(y))\]
Symbols: \(p(x)p(y)\) = product of marginals (independence assumption).
Intuition: Mutual information measures how much the joint distribution differs from independence. Zero KL = independence; high KL = strong dependence. This is why MI ≥ 0.

Q: What is the data processing inequality?
A: \[X \to Y \to Z \implies I(X;Z) \leq I(X;Y)\]
Symbols: Markov chain X→Y→Z; processing Y to get Z can't add information about X.
Intuition: You can never increase information about X by processing it further. Any transformation discards or preserves, but never creates, information. Key in feature extraction theory.

Q: What is the channel capacity?
A: \[C = \max_{p(x)} I(X;Y)\]
Symbols: \(C\) = maximum bits per use of a channel; maximized over input distributions.
Intuition: Shannon's noisy channel coding theorem: C is the maximum reliable transmission rate. Below C: error-free communication possible; above C: errors unavoidable.

Q: What is conditional mutual information?
A: \[I(X;Y|Z) = H(X|Z) - H(X|Y,Z)\]
Symbols: \(I(X;Y|Z) \geq 0\); equals 0 iff X⊥Y|Z.
Intuition: Information shared between X and Y after knowing Z. Used in graphical models to test conditional independence and in feature selection (e.g., CMIM algorithm).

Q: What is the Kullback-Leibler (KL) divergence?
A: \[D_{\mathrm{KL}}(P\|Q) = \sum_x p(x)\log\frac{p(x)}{q(x)}\]
Symbols: \(P\) = true distribution; \(Q\) = approximate distribution; result in nats or bits.
Intuition: Extra bits needed to code samples from P using a code optimized for Q. Measures information lost by approximating P with Q.

Q: Is KL divergence symmetric?
A: \[D_{\mathrm{KL}}(P\|Q) \neq D_{\mathrm{KL}}(Q\|P) \text{ in general}\]
Symbols: \(D_{\mathrm{KL}}(P\|Q)\) = forward KL; \(D_{\mathrm{KL}}(Q\|P)\) = reverse KL.
Intuition: KL divergence is not a distance metric (no symmetry). Forward KL (P‖Q): Q must cover all of P ('mean-seeking'). Reverse KL (Q‖P): Q concentrates on modes of P ('mode-seeking').

Q: Why is KL divergence always non-negative?
A: \[D_{\mathrm{KL}}(P\|Q) \geq 0, \text{ with equality iff } P=Q\]
Symbols: Follows from Jensen's inequality: \(-\log\) is convex.
Intuition: By Jensen's: \(\mathbb{E}_P[-\log(Q/P)] \geq -\log\mathbb{E}_P[Q/P] = -\log 1 = 0\). Extra code length is always non-negative — you can't do better than the true distribution.

Q: What is the forward vs reverse KL divergence in variational inference?
A: \[D_{\mathrm{KL}}(q\|p) = \mathbb{E}_q\!\left[\log\frac{q(z)}{p(z|x)}\right]\]
Symbols: Reverse KL: mode-seeking, q concentrates on a mode of p.
Intuition: Minimizing reverse KL (q‖p) makes q concentrate on the highest peak of p — can miss multiple modes (under-dispersed). VAEs minimize this. Forward KL (p‖q) makes q cover all of p — can be over-dispersed.

Q: What is the KL divergence between two Gaussians?
A: \[D_{\mathrm{KL}}(\mathcal{N}_1\|\mathcal{N}_2) = \log\frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2+(\mu_1-\mu_2)^2}{2\sigma_2^2} - \frac{1}{2}\]
Symbols: \(\mathcal{N}_1=\mathcal{N}(\mu_1,\sigma_1^2)\); \(\mathcal{N}_2=\mathcal{N}(\mu_2,\sigma_2^2)\).
Intuition: Has an analytic closed form. Used extensively in VAE loss: KL between posterior q(z|x) and prior p(z)=N(0,I).

Q: What is the KL divergence from q(z|x)=N(μ,σ²) to p(z)=N(0,I) in a VAE?
A: \[D_{\mathrm{KL}}(q\|p) = -\frac{1}{2}\sum_{j=1}^d \left(1 + \log\sigma_j^2 - \mu_j^2 - \sigma_j^2\right)\]
Symbols: \(d\) = latent dimension; \(\mu_j, \sigma_j^2\) = variational parameters per dimension.
Intuition: Closed-form regularization term in VAE training. Pushes posterior toward the standard Gaussian prior — prevents the encoder from ignoring the prior.

Q: What is cross-entropy?
A: \[H(P,Q) = -\sum_x p(x)\log q(x)\]
Symbols: \(P\) = true distribution; \(Q\) = model distribution; result = expected code length under Q.
Intuition: Average number of bits needed to encode events from P using code optimized for Q. Always ≥ H(P). Minimizing cross-entropy is the standard classification training objective.

Q: How does cross-entropy relate to entropy and KL divergence?
A: \[H(P,Q) = H(P) + D_{\mathrm{KL}}(P\|Q)\]
Symbols: \(H(P)\) = entropy of true distribution (constant w.r.t. model); \(D_{\mathrm{KL}}\) = extra cost.
Intuition: Cross-entropy = true entropy + KL divergence. Since H(P) is constant (not controlled by model), minimizing cross-entropy is equivalent to minimizing KL(P‖Q).

Q: What is the cross-entropy loss for multi-class classification?
A: \[\mathcal{L} = -\sum_{k=1}^K y_k \log \hat{y}_k\]
Symbols: \(y_k \in \{0,1\}\) = one-hot label; \(\hat{y}_k\) = predicted probability for class k.
Intuition: Only the log-probability of the true class contributes. Equivalent to negative log-likelihood of Categorical distribution. With softmax outputs, gradient is simply ŷ - y.

Q: What is the binary cross-entropy loss?
A: \[\mathcal{L} = -[y\log\hat{y} + (1-y)\log(1-\hat{y})]\]
Symbols: \(y \in \{0,1\}\) = binary label; \(\hat{y} = \sigma(z) \in (0,1)\) = sigmoid output.
Intuition: For binary classification: if y=1, penalizes low ŷ; if y=0, penalizes high ŷ. Negative log-likelihood of Bernoulli distribution. Gradient: ŷ - y.

Q: Why does cross-entropy penalize wrong confident predictions so heavily?
A: \[\mathcal{L} = -\log\hat{y}_{\mathrm{true}}; \quad \lim_{\hat{y}\to 0}-\log\hat{y} = \infty\]
Symbols: \(\hat{y}_{\mathrm{true}}\) = predicted probability for correct class.
Intuition: Log blows up near zero. A confident wrong prediction (true class probability → 0) gives infinite loss. This strongly penalizes overconfident mistakes — more so than squared error.

Q: What is the cross-entropy between the empirical data distribution and the model?
A: \[H(p_{\mathrm{data}},p_\theta) = -\frac{1}{N}\sum_{i=1}^N \log p_\theta(x_i)\]
Symbols: \(p_{\mathrm{data}}\) = uniform over training set; \(p_\theta\) = model.
Intuition: The average negative log-likelihood — the standard training loss. Minimizing this is exactly MLE and exactly minimizing KL from data distribution to model.

Q: What is the Jensen-Shannon (JS) divergence?
A: \[\mathrm{JSD}(P\|Q) = \frac{1}{2}D_{\mathrm{KL}}(P\|M) + \frac{1}{2}D_{\mathrm{KL}}(Q\|M), \quad M = \frac{P+Q}{2}\]
Symbols: \(M\) = mixture distribution; \(\mathrm{JSD} \in [0, \log 2]\).
Intuition: Symmetrized version of KL. Bounded and symmetric — unlike KL. \(\sqrt{\mathrm{JSD}}\) is a proper metric. Used in GANs: the original GAN objective minimizes JSD between real and generated distributions.

Q: Why does the original GAN training objective minimize JS divergence?
A: \[\min_G \max_D V(D,G) = \mathbb{E}_p[\log D(x)] + \mathbb{E}_q[\log(1-D(G(z)))] \equiv -\log 4 + 2\,\mathrm{JSD}(p\|q)\]
Symbols: \(D\) = discriminator; \(G\) = generator; \(p\) = real; \(q\) = generated.
Intuition: At the optimal discriminator, GAN training is equivalent to minimizing JSD. JS divergence is zero only when real and generated distributions match exactly.

Q: What is information gain in decision trees?
A: \[\mathrm{IG}(S,A) = H(S) - \sum_{v \in \mathrm{vals}(A)}\frac{|S_v|}{|S|}H(S_v)\]
Symbols: \(S\) = dataset; \(A\) = attribute to split on; \(S_v\) = subset with attribute value v.
Intuition: Reduction in entropy after splitting on attribute A. Decision trees greedily choose the split with highest information gain — equivalent to minimizing weighted average child entropy.

Q: What is the Gini impurity?
A: \[\mathrm{Gini}(p) = 1 - \sum_{k=1}^K p_k^2 = \sum_{k \neq j} p_k p_j\]
Symbols: \(p_k\) = fraction of class k; \(\mathrm{Gini}=0\) for pure node.
Intuition: Probability that two randomly drawn samples have different classes. Approximates entropy but cheaper to compute (no log). Used by CART algorithm instead of information gain.

Q: What is the relationship between cross-entropy and perplexity in language models?
A: \[\mathrm{PPL} = 2^{H(P,Q)} = 2^{-\frac{1}{N}\sum_{i=1}^N \log_2 p(w_i|w_{1:i-1})}\]
Symbols: \(\mathrm{PPL}\) = perplexity; \(H(P,Q)\) = cross-entropy in bits.
Intuition: Perplexity = 2^(cross-entropy) — the effective vocabulary size the model is 'confused' between at each step. Lower = better. A model with PPL=10 is as uncertain as if choosing uniformly from 10 words.

Q: What is the mutual information between input and output in a neural network layer?
A: \[I(X;Z) = H(Z) - H(Z|X)\]
Symbols: \(Z\) = layer output; \(H(Z|X)=0\) for deterministic layers (output determined by input).
Intuition: The information bottleneck framework: good representations compress X (minimize I(X;Z)) while preserving label information (maximize I(Z;Y)). Relevant to understanding what layers learn.

Q: What is the information bottleneck principle?
A: \[\min_{p(z|x)} I(X;Z) - \beta I(Z;Y)\]
Symbols: \(Z\) = compressed representation; \(Y\) = target; \(\beta\) = tradeoff; minimize compression while preserving prediction.
Intuition: Find the minimal sufficient statistic for predicting Y from X. The tradeoff parameter β controls compression vs. accuracy. Theoretical lens on what representations DNNs learn.

Q: What is the entropy rate of a stationary process?
A: \[\mathcal{H} = \lim_{N\to\infty} \frac{1}{N}H(X_1,\ldots,X_N)\]
Symbols: \(\mathcal{H}\) = entropy rate; limit of average entropy per symbol.
Intuition: How much new information each new symbol adds, on average, in a long sequence. Language has entropy rate ≈ 1 bit/character (Shannon's experiment). Language models are optimized to minimize this.

Q: What is total correlation (multi-information)?
A: \[\mathrm{TC}(X_1,\ldots,X_n) = \sum_i H(X_i) - H(X_1,\ldots,X_n)\]
Symbols: \(\sum_i H(X_i)\) = sum of marginal entropies; \(H(\cdot)\) = joint entropy.
Intuition: Generalization of mutual information to n variables. Measures total statistical dependence. Zero iff all variables are mutually independent. Used in independent component analysis (ICA).

Q: What is Fano's inequality?
A: \[H(X|\hat{X}) \leq H_b(P_e) + P_e \log(|\mathcal{X}|-1)\]
Symbols: \(P_e\) = probability of error; \(H_b\) = binary entropy; \(\hat{X}\) = estimate of X.
Intuition: Lower bounds the error probability: if H(X|X̂) is large, prediction must be uncertain. Used to prove converse capacity theorems and understand fundamental limits of classification.

Q: What is the relationship between entropy and compression (Shannon's source coding theorem)?
A: \[\text{Min avg code length} \geq H(X)\]
Symbols: \(H(X)\) = entropy in bits; no lossless code can do better on average.
Intuition: Entropy is the fundamental limit of lossless compression. Optimal codes (like Huffman) achieve ≈H(X) bits per symbol. Cannot compress a source below its entropy without loss.

Q: What is Huffman coding and how does entropy relate to it?
A: \[\bar{L}_{\mathrm{Huffman}} < H(X) + 1\]
Symbols: \(\bar{L}\) = average code length; Huffman achieves within 1 bit of entropy.
Intuition: Huffman assigns shorter codes to frequent symbols. Average length is between H(X) and H(X)+1 bits. An optimal prefix-free code. Achieves Shannon's bound up to 1 bit.

Q: What is the pointwise mutual information (PMI)?
A: \[\mathrm{PMI}(x,y) = \log\frac{p(x,y)}{p(x)p(y)}\]
Symbols: \(p(x,y)\) = joint; \(p(x)p(y)\) = independence baseline; positive PMI = co-occur more than chance.
Intuition: Measures how much more (or less) two events co-occur compared to if they were independent. Foundation of word2vec (PPMI matrices) and word association measures in NLP.

Q: What is the R'enyi entropy?
A: \[H_\alpha(X) = \frac{1}{1-\alpha}\log\sum_x p(x)^\alpha\]
Symbols: \(\alpha \geq 0\) = order parameter; \(\alpha \to 1\) → Shannon entropy; \(\alpha=2\) → collision entropy.
Intuition: A family of entropy measures with different sensitivity to rare events. Lower α = more weight on rare events; higher α = more weight on common events. Includes Shannon as a special case.

Q: What is the R'enyi divergence?
A: \[D_\alpha(P\|Q) = \frac{1}{\alpha-1}\log\sum_x p(x)^\alpha q(x)^{1-\alpha}\]
Symbols: \(\alpha \to 1\) → KL divergence; \(D_\alpha \geq 0\).
Intuition: Generalizes KL divergence. Different orders emphasize different parts of the distributions. Used in differential privacy (α=∞ → max divergence) and theoretical ML analysis.

Q: What is the relationship between cross-entropy and the KL divergence in practice?
A: \[\mathcal{L}_{\mathrm{CE}} = H(y, \hat{y}) = H(y) + D_{\mathrm{KL}}(y\|\hat{y})\]
Symbols: \(H(y)\) = fixed entropy of true labels (0 for one-hot); so \(\mathcal{L}_{\mathrm{CE}} = D_{\mathrm{KL}}\).
Intuition: For one-hot labels, H(y)=0, so cross-entropy loss exactly equals KL divergence. Training a classifier with cross-entropy = minimizing KL divergence from predictions to ground truth.

Q: What is label smoothing and how does it affect the cross-entropy?
A: \[y_k^{\mathrm{smooth}} = (1-\epsilon)y_k + \frac{\epsilon}{K}, \quad \mathcal{L} = -\sum_k y_k^{\mathrm{smooth}}\log\hat{y}_k\]
Symbols: \(\epsilon\) = smoothing factor; \(K\) = number of classes; reduces overconfidence.
Intuition: Replaces hard 0/1 labels with soft distributions. Prevents the model from becoming overconfident (predicted logits going to ±∞). Acts as a regularizer, improving calibration.

Q: What is the normalized mutual information?
A: \[\mathrm{NMI}(X,Y) = \frac{I(X;Y)}{\sqrt{H(X)H(Y)}} \in [0,1]\]
Symbols: \(I(X;Y)\) = mutual information; normalized to [0,1] by geometric mean of entropies.
Intuition: Normalized version of MI suitable for comparing different pairs of variables. 0 = independent; 1 = perfectly dependent. Used to evaluate clustering quality (compare cluster labels to true labels).

Q: What is an objective (loss) function in ML optimization?
A: \[\min_{\theta} \mathcal{L}(\theta) = \frac{1}{N}\sum_{i=1}^N \ell(f_\theta(x_i), y_i)\]
Symbols: \(\theta\) = parameters; \(\ell\) = per-sample loss; \(f_\theta\) = model.
Intuition: The scalar quantity we minimize to train the model. Average per-sample loss over the training set. Different tasks use different loss functions (MSE, CE, hinge).

Q: What is the difference between a local and global minimum?
A: \[\text{Local: } \mathcal{L}(\theta^*) \leq \mathcal{L}(\theta) \text{ for } \|\theta-\theta^*\|<\delta; \quad \text{Global: for all } \theta\]
Symbols: \(\delta\) = small neighborhood radius.
Intuition: A local minimum is better than all nearby points but might not be the best overall. Deep nets have many local minima, but in practice most are near-equivalent (flat minima). Global min guarantees exist only for convex problems.

Q: What is a saddle point and how does it differ from a local minimum?
A: \[\nabla\mathcal{L}(\theta^*) = 0, \quad \exists\, v: v^\top H v > 0 \text{ and } \exists\, u: u^\top H u < 0\]
Symbols: \(H\) = Hessian; indefinite means curvature differs by direction.
Intuition: Gradient is zero but it's not a minimum — some directions go down. Saddle points are far more common than local minima in high dimensions. Noise in SGD helps escape them.

Q: What is a convex optimization problem?
A: \[\min_\theta \mathcal{L}(\theta) \text{ where } \mathcal{L}(\lambda\theta_1+(1-\lambda)\theta_2) \leq \lambda\mathcal{L}(\theta_1)+(1-\lambda)\mathcal{L}(\theta_2)\]
Symbols: \(\lambda \in [0,1]\); any local minimum is a global minimum for convex problems.
Intuition: The loss landscape has no valleys — any downhill step leads toward the global optimum. Logistic regression is convex; neural networks are generally not.

Q: What is a strongly convex function?
A: \[\mathcal{L}(\theta_2) \geq \mathcal{L}(\theta_1) + \nabla\mathcal{L}(\theta_1)^\top(\theta_2-\theta_1) + \frac{m}{2}\|\theta_2-\theta_1\|^2\]
Symbols: \(m > 0\) = strong convexity constant; quadratic lower bound.
Intuition: Stronger than convexity — the function grows at least quadratically away from the minimum. Guarantees a unique global minimum and faster convergence (linear convergence rate for GD).

Q: What is an L-smooth function?
A: \[\|\nabla\mathcal{L}(\theta_1) - \nabla\mathcal{L}(\theta_2)\| \leq L\|\theta_1-\theta_2\|\]
Symbols: \(L\) = Lipschitz constant of the gradient; \(H \preceq LI\) (Hessian bounded).
Intuition: The gradient doesn't change too rapidly. For L-smooth functions, GD with learning rate η ≤ 1/L is guaranteed to decrease the loss each step. Relates to the maximum safe learning rate.

Q: What is the convergence rate of gradient descent for L-smooth convex functions?
A: \[\mathcal{L}(\theta_T) - \mathcal{L}(\theta^*) \leq \frac{L\|\theta_0 - \theta^*\|^2}{2T}\]
Symbols: \(T\) = iterations; \(\theta^*\) = optimal parameters; \(L\) = smoothness.
Intuition: The loss excess decreases as O(1/T) — sublinear convergence. For strongly convex functions it's O(ρ^T) (linear/geometric convergence), which is much faster.

Q: What is the gradient descent update rule?
A: \[\theta_{t+1} = \theta_t - \eta \nabla_\theta \mathcal{L}(\theta_t)\]
Symbols: \(\eta\) = learning rate; \(\nabla_\theta \mathcal{L}\) = gradient of loss w.r.t. \(\theta\).
Intuition: Move parameters opposite to the gradient — downhill on the loss surface. Small η: safe but slow. Large η: fast but can diverge.

Q: What is the effect of learning rate on gradient descent?
A: \[\text{Too large: } \mathcal{L} \text{ diverges; too small: slow convergence; optimal: } \eta \approx \frac{1}{L}\]
Symbols: \(L\) = Lipschitz constant of gradient; \(\eta_{\mathrm{opt}} = 1/L\) for quadratics.
Intuition: Learning rate is the most important hyperparameter. Too large = gradient descent overshoots and diverges. Too small = too many steps needed. Optimal ≈ inverse of curvature.

Q: What is the loss decrease guarantee for one gradient descent step?
A: \[\mathcal{L}(\theta - \eta\nabla\mathcal{L}) \leq \mathcal{L}(\theta) - \eta\left(1 - \frac{\eta L}{2}\right)\|\nabla\mathcal{L}\|^2\]
Symbols: \(L\) = smoothness constant; decreases if \(\eta < 2/L\).
Intuition: For L-smooth functions, each step decreases the loss proportional to the squared gradient norm. This holds as long as the step size is small enough.

Q: What is Stochastic Gradient Descent (SGD)?
A: \[\theta_{t+1} = \theta_t - \eta_t \nabla_\theta \ell(f_\theta(x_i), y_i)\]
Symbols: \((x_i, y_i)\) = single randomly sampled training example; noisy gradient estimate.
Intuition: Instead of computing the full gradient over all N examples, use a single random example. Much cheaper per step (O(1) vs O(N)), converges in expectation to a minimum.

Q: What is mini-batch SGD?
A: \[\theta_{t+1} = \theta_t - \frac{\eta}{B}\sum_{i \in \mathcal{B}} \nabla_\theta \ell_i(\theta_t)\]
Symbols: \(\mathcal{B}\) = mini-batch of B examples; \(\eta/B\) = scaled learning rate.
Intuition: Compromise between full GD (accurate but slow per step) and SGD (noisy but fast). Batch size B balances noise level and parallelism. Common choice: B=32 to 512.

Q: Why does noise in SGD sometimes help?
A: \[\theta_{t+1} = \theta_t - \eta\nabla\mathcal{L}_{\mathcal{B}}(\theta_t) + \text{noise}\]
Symbols: Stochastic noise ∝ gradient variance / batch size.
Intuition: Gradient noise acts as implicit regularization — helps escape sharp local minima and saddle points. Models trained with SGD often generalize better than those trained with exact gradient methods (implicit bias toward flat minima).

Q: What is the bias-variance tradeoff of gradient estimators?
A: \[\text{Full GD: zero variance, but O(N) cost; SGD: high variance, O(1) cost}\]
Symbols: Variance of mini-batch gradient ∝ \(\sigma^2_{\mathrm{grad}}/B\).
Intuition: Larger batches give lower variance gradient estimates but cost more per step. Smaller batches allow more steps per second but noisier updates. Variance decreases as 1/B.

Q: What is gradient descent with momentum?
A: \[v_{t+1} = \beta v_t + \nabla\mathcal{L}(\theta_t), \quad \theta_{t+1} = \theta_t - \eta v_{t+1}\]
Symbols: \(v_t\) = velocity (exponential moving avg of gradients); \(\beta \in [0,1)\) = momentum coefficient (e.g., 0.9).
Intuition: Accumulates gradient history — builds up speed in consistent directions, damps oscillations in oscillating directions. Accelerates convergence in narrow valleys.

Q: What is the intuition behind momentum in optimization?
A: \[v = \sum_{k=0}^\infty \beta^k \nabla\mathcal{L}_{t-k} \approx \frac{\nabla\mathcal{L}}{1-\beta} \text{ if grad constant}\]
Symbols: \(\beta^k\) = exponential decay of past gradients; effective LR ≈ \(\eta/(1-\beta)\).
Intuition: Like a ball rolling down a hill that builds momentum. Consistent gradient directions accumulate, amplifying effective step size. β=0.9 means gradient direction must be consistent for ~10 steps to get full momentum.

Q: What is Nesterov momentum?
A: \[v_{t+1} = \beta v_t + \nabla\mathcal{L}(\theta_t - \beta\eta v_t), \quad \theta_{t+1} = \theta_t - \eta v_{t+1}\]
Symbols: Gradient evaluated at the 'look-ahead' position \(\theta_t - \beta\eta v_t\).
Intuition: Computes gradient at where we're about to be (after momentum step), not where we are. More 'anticipatory' — slightly faster convergence than standard momentum, especially near minima.

Q: What is AdaGrad?
A: \[G_t = G_{t-1} + g_t^2, \quad \theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{G_t}+\epsilon}g_t\]
Symbols: \(g_t = \nabla\mathcal{L}(\theta_t)\); \(G_t\) = accumulated squared gradients (element-wise); \(\epsilon\) = small constant for stability.
Intuition: Adapts learning rate per parameter: parameters with large historical gradients get smaller LR; sparse parameters (small historical gradient) get larger LR. Great for sparse features (NLP). Problem: LR keeps shrinking to 0.

Q: Why does AdaGrad's learning rate monotonically decrease?
A: \[G_t = \sum_{k=1}^t g_k^2 \nearrow \infty \implies \frac{\eta}{\sqrt{G_t}} \searrow 0\]
Symbols: \(G_t\) accumulates all past squared gradients monotonically.
Intuition: AdaGrad keeps adding to G, so the effective learning rate eventually becomes zero — training stops prematurely. RMSprop and Adam fix this with exponential moving averages.

Q: What is RMSprop?
A: \[v_t = \rho v_{t-1} + (1-\rho)g_t^2, \quad \theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{v_t}+\epsilon}g_t\]
Symbols: \(v_t\) = exponential moving avg of squared gradients; \(\rho \approx 0.9\) = decay rate.
Intuition: Fixes AdaGrad's diminishing LR by using an exponential moving average instead of sum — only recent gradient magnitudes matter. Non-stationary friendly; widely used in RNNs.

Q: What is the effective learning rate in RMSprop?
A: \[\eta_{\mathrm{eff}} = \frac{\eta}{\sqrt{v_t}} \approx \frac{\eta}{\text{RMS}(g_t)}\]
Symbols: \(\text{RMS}(g_t)\) = root mean square of recent gradients.
Intuition: Each parameter gets a learning rate inversely proportional to the RMS of its recent gradients. Large gradient direction → small step. Small gradient direction → large step. Normalizes gradients by their recent scale.

Q: What is the Adam optimizer?
A: \[m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t, \quad v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2\]
Symbols: \(m_t\) = 1st moment (mean); \(v_t\) = 2nd moment (uncentered variance); \(\beta_1 \approx 0.9\); \(\beta_2 \approx 0.999\).
Intuition: Combines momentum (1st moment) and RMSprop (2nd moment). Each parameter gets an adaptive learning rate scaled by the ratio of estimated mean to estimated RMS.

Q: What is the bias correction in Adam?
A: \[\hat{m}_t = \frac{m_t}{1-\beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1-\beta_2^t}\]
Symbols: \(1-\beta_k^t \to 0\) initially (small t) → correction makes early estimates accurate.
Intuition: At step 1, m₁ = (1-β₁)g₁ is tiny because m₀=0. Bias correction scales it up by 1/(1-β₁) to correct for the zero initialization. Without this, early steps would use underestimated moments.

Q: What is the Adam parameter update rule?
A: \[\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v}_t}+\epsilon}\hat{m}_t\]
Symbols: \(\hat{m}_t\) = bias-corrected 1st moment; \(\hat{v}_t\) = bias-corrected 2nd moment; \(\epsilon \approx 10^{-8}\).
Intuition: Step size ≈ η × (gradient direction) / (gradient scale). Approximately unit step size per parameter. Default hyperparameters (η=0.001, β₁=0.9, β₂=0.999) work well out-of-the-box for most tasks.

Q: What is the effective learning rate in Adam?
A: \[\eta_{\mathrm{eff}} \approx \eta \frac{\hat{m}_t}{\sqrt{\hat{v}_t}} \approx \eta \cdot \text{sign}(g_t) \text{ (heuristically)}\]
Symbols: \(\hat{m}_t/\sqrt{\hat{v}_t}\) ≈ signal-to-noise ratio of gradient; bounded by ±1.
Intuition: Adam normalizes gradient by its own scale — approximately equivalent to taking unit steps in the gradient direction. This makes it robust to gradient scale but can hurt generalization compared to SGD.

Q: What is AdamW?
A: \[\theta_{t+1} = \theta_t - \eta\left(\frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon} + \lambda\theta_t\right)\]
Symbols: \(\lambda\) = weight decay coefficient; weight decay applied directly, not through the gradient.
Intuition: Adam + decoupled weight decay. In original Adam, L2 reg is divided by √v̂ (adaptive scaling), which weakens its effect. AdamW applies weight decay directly — better generalization. Default in modern transformer training.

Q: What is cosine annealing learning rate schedule?
A: \[\eta_t = \eta_{\min} + \frac{1}{2}(\eta_{\max}-\eta_{\min})\left(1 + \cos\frac{\pi t}{T}\right)\]
Symbols: \(T\) = total steps; \(\eta_{\max}, \eta_{\min}\) = max and min learning rates.
Intuition: Smoothly decays the learning rate following a cosine curve. Starts high (fast progress), ends near zero (fine-tuning). Widely used with transformer training.

Q: What is learning rate warmup?
A: \[\eta_t = \eta_{\max} \cdot \frac{t}{T_{\mathrm{warm}}} \text{ for } t \leq T_{\mathrm{warm}}\]
Symbols: \(T_{\mathrm{warm}}\) = warmup steps; linearly increases LR from 0 to η_max.
Intuition: Avoids large steps at the start when parameters are randomly initialized (moments in Adam are unreliable early on). Warmup + cosine decay is the standard transformer training schedule.

Q: What is step decay learning rate schedule?
A: \[\eta_t = \eta_0 \cdot \gamma^{\lfloor t/s \rfloor}\]
Symbols: \(\gamma < 1\) = decay factor (e.g., 0.1); \(s\) = step size (every s steps LR drops).
Intuition: Multiplies learning rate by γ every s steps. Simple and widely used in image classification (e.g., ResNet training: divide by 10 at epochs 100 and 150).

Q: What is the 1-cycle learning rate policy?
A: \[\eta: 0 \to \eta_{\max} \to 0 \text{ (linear warmup + cosine decay in one cycle)}\]
Symbols: Used with SGD+momentum; \(\eta_{\max}\) found by LR range test.
Intuition: Start small, peak in the middle, end near zero. One full cycle trains the model faster than multi-stage schedules. The high LR in the middle helps escape sharp minima.

Q: What is Newton's method for optimization?
A: \[\theta_{t+1} = \theta_t - [H(\theta_t)]^{-1}\nabla\mathcal{L}(\theta_t)\]
Symbols: \(H = \nabla^2\mathcal{L}\) = Hessian; \(H^{-1}\) = inverse Hessian.
Intuition: Uses curvature (Hessian) to scale the gradient — takes a large step in flat directions and small step in steep directions. Quadratic convergence near the optimum. Too expensive for DL (O(n²) memory, O(n³) inversion).

Q: Why is Newton's method impractical for large neural networks?
A: \[H \in \mathbb{R}^{n \times n}, \quad n = \text{number of parameters (millions)}\]
Symbols: \(H^{-1}\) requires \(O(n^2)\) memory and \(O(n^3)\) computation.
Intuition: A network with 1M parameters has a 1M×1M Hessian — impossible to store or invert. Quasi-Newton methods (L-BFGS) approximate H^{-1} with low-rank updates. Used for small-scale problems.

Q: What is L-BFGS?
A: \[H_t^{-1} \approx \text{limited-memory rank-2 updates from recent } (s_k,y_k) \text{ pairs}\]
Symbols: \(s_k = \theta_k-\theta_{k-1}\); \(y_k = \nabla\mathcal{L}_k - \nabla\mathcal{L}_{k-1}\); stores last m pairs (typically m=20).
Intuition: Approximates the inverse Hessian using only the last m gradient differences — O(mn) memory instead of O(n²). Used for full-batch optimization in small models and in scientific computing.

Q: What is a constrained optimization problem?
A: \[\min_\theta f(\theta) \quad \text{s.t.} \quad g_i(\theta) = 0, \; h_j(\theta) \leq 0\]
Symbols: \(g_i\) = equality constraints; \(h_j\) = inequality constraints.
Intuition: Find the minimum of f subject to constraints on θ. Methods: Lagrange multipliers (equality), KKT (inequality), projected gradient (simple constraints like parameter bounds).

Q: What is the Lagrangian for constrained optimization?
A: \[\mathcal{L}(\theta,\lambda,\mu) = f(\theta) + \sum_i \lambda_i g_i(\theta) + \sum_j \mu_j h_j(\theta)\]
Symbols: \(\lambda_i \in \mathbb{R}\) = equality multipliers; \(\mu_j \geq 0\) = inequality multipliers.
Intuition: Augments the objective with penalties for constraint violation. At the optimum, stationarity ∇L=0 gives the KKT conditions. Dual variables λ,μ measure constraint tightness.

Q: What is the Lagrangian dual function?
A: \[d(\lambda,\mu) = \min_\theta \mathcal{L}(\theta,\lambda,\mu) \leq f(\theta^*)\]
Symbols: \(d(\lambda,\mu)\) = dual function (concave); \(\theta^*\) = primal optimal.
Intuition: The dual function is always a lower bound on the primal optimal. Strong duality (Slater's condition): dual max = primal min. Used in SVMs — solving the dual is often easier.

Q: What is strong duality and Slater's condition?
A: \[\text{Strong duality: } \min_\theta f = \max_{\mu\geq 0} d(\lambda,\mu) \text{ when Slater's holds}\]
Symbols: Slater's condition: ∃ strictly feasible point where all inequality constraints are strict.
Intuition: The duality gap is zero. Common in convex problems. Allows solving the dual (often simpler) to get the primal solution. Key in SVM dual formulation.

Q: What is projected gradient descent for constrained problems?
A: \[\theta_{t+1} = \Pi_C(\theta_t - \eta\nabla f(\theta_t))\]
Symbols: \(\Pi_C\) = projection onto feasible set C; maps back into C after gradient step.
Intuition: Take an unconstrained gradient step, then project back onto the feasible set. Simple and effective for box constraints (min/max parameter bounds) and sphere constraints.

Q: What is the proximal operator?
A: \[\mathrm{prox}_{\eta g}(\theta) = \arg\min_u \left(\frac{1}{2}\|u-\theta\|^2 + \eta g(u)\right)\]
Symbols: \(g\) = regularizer (possibly non-smooth); \(\eta\) = step size.
Intuition: Finds the point closest to θ while also minimizing g. For g = λ|·|₁: proximal operator is soft-thresholding (used in LASSO). Handles non-smooth regularization.

Q: What is the soft-thresholding operator for L1 regularization?
A: \[\mathrm{prox}_{\eta\lambda\|\cdot\|_1}(\theta_j) = \mathrm{sign}(\theta_j)\max(|\theta_j|-\eta\lambda, 0)\]
Symbols: \(\lambda\) = L1 penalty; \(\eta\) = step size; shrinks small weights to exactly zero.
Intuition: The proximal operator for L1 shrinks each weight toward zero by η λ, setting it to zero if it would overshoot. This is what makes L1 regularization produce sparse solutions.

Q: What is grid search for hyperparameter optimization?
A: \[\theta^* = \arg\min_{\theta \in \Theta_{\mathrm{grid}}} \mathcal{L}_{\mathrm{val}}(f_\theta)\]
Symbols: \(\Theta_{\mathrm{grid}}\) = Cartesian product of discrete hyperparameter values.
Intuition: Exhaustively try all combinations. Simple but exponential in number of hyperparameters. Random search (Bergstra & Bengio 2012) is usually better — searches the same space more efficiently.

Q: What is Bayesian optimization for hyperparameters?
A: \[\alpha_t(\theta) = \mathbb{E}[\mathcal{L}^+ | \theta, \mathcal{D}_t], \quad \theta_t = \arg\max_\theta \alpha_t(\theta)\]
Symbols: \(\alpha_t\) = acquisition function; \(\mathcal{D}_t\) = past (θ, L) evaluations; uses Gaussian process surrogate.
Intuition: Builds a probabilistic model of the loss as a function of hyperparameters, then selects the next point to evaluate using an acquisition function (balancing exploration and exploitation).

Q: What is a flat minimum and why is it desirable?
A: \[\lambda_{\max}(H(\theta^*)) \text{ small} \implies \text{flat minimum; better generalization}\]
Symbols: \(\lambda_{\max}(H)\) = largest Hessian eigenvalue at minimum; sharpness measure.
Intuition: Flat minima generalize better — small changes in parameters don't increase loss much. Sharp minima are sensitive to small perturbations (poor generalization). SGD with large batch tends toward sharp minima.

Q: What is sharpness-aware minimization (SAM)?
A: \[\min_\theta \max_{\|\epsilon\|\leq\rho} \mathcal{L}(\theta+\epsilon) \approx \min_\theta \mathcal{L}(\theta + \rho\nabla\mathcal{L}/\|\nabla\mathcal{L}\|)\]
Symbols: \(\rho\) = perturbation radius; minimizes worst-case loss in a neighborhood.
Intuition: Explicitly seeks flat minima by minimizing the maximum loss in a ball around θ. Requires two gradient evaluations per step but improves generalization significantly.

Q: What is gradient clipping and why is it used?
A: \[\mathbf{g} \leftarrow \frac{\tau\,\mathbf{g}}{\max(\|\mathbf{g}\|, \tau)}\]
Symbols: \(\tau\) = clipping threshold; \(\|\mathbf{g}\|\) = gradient norm.
Intuition: Prevents exploding gradients by rescaling the gradient to have norm at most τ. Essential for RNNs and transformers on long sequences where gradients can grow exponentially through time.

Q: What is the loss surface of a neural network typically like?
A: \[\text{Many saddle points, few isolated local minima, many equivalent global minima}\]
Symbols: High-dimensional landscape; Hessian has many near-zero eigenvalues (flat directions).
Intuition: Deep networks have exponentially many equivalent near-optimal solutions (due to symmetries). Most critical points are saddle points, not local minima. This makes training surprisingly tractable.

Q: What is the gradient of the L2 regularized loss?
A: \[\nabla_\theta \mathcal{L}_{\mathrm{reg}} = \nabla_\theta \mathcal{L} + \lambda\theta\]
Symbols: \(\lambda\) = regularization strength; extra \(\lambda\theta\) term pulls weights toward 0.
Intuition: L2 regularization adds a gradient that always pulls weights toward zero. With SGD this is equivalent to weight decay: \(\theta \leftarrow (1-\lambda\eta)\theta - \eta\nabla\mathcal{L}\).

Q: What is the convergence condition for SGD on convex problems?
A: \[\sum_{t=1}^\infty \eta_t = \infty, \quad \sum_{t=1}^\infty \eta_t^2 < \infty \quad (\text{e.g., } \eta_t = 1/t)\]
Symbols: Robbins-Monro conditions on the learning rate schedule.
Intuition: First condition ensures SGD makes enough total progress to reach the optimum. Second ensures steps decrease fast enough so noise doesn't prevent convergence. Decaying LR is necessary for convergence of SGD.

Q: What is the difference between first-order and second-order optimization methods?
A: \[\text{1st order: uses } \nabla\mathcal{L} \in \mathbb{R}^n; \quad \text{2nd order: uses } H = \nabla^2\mathcal{L} \in \mathbb{R}^{n\times n}\]
Symbols: \(n\) = number of parameters; 2nd order has \(O(n^2)\) memory.
Intuition: First-order: GD, SGD, Adam — only need gradients. Second-order: Newton, L-BFGS — use curvature for faster convergence but much higher cost. Practical DL uses first-order methods.

Q: What is a learning rate finder?
A: \[\eta^* = \arg\max_\eta \left(-\frac{d\mathcal{L}}{d\log\eta}\right)\]
Symbols: Find LR where loss decreases fastest (steepest negative slope on LR vs. loss curve).
Intuition: Run training with exponentially increasing LR for one epoch. Plot LR vs. loss and pick the LR where loss decreases most steeply (just before it starts to diverge). Proposed by Leslie Smith (2015).

Q: What is gradient variance and how does batch size affect it?
A: \[\mathrm{Var}[\hat{g}] = \frac{\mathrm{Var}[g_i]}{B}\]
Symbols: \(B\) = batch size; \(g_i\) = single-sample gradient; variance decreases as \(1/B\).
Intuition: Averaging B gradients reduces variance by a factor of B. Doubling batch size halves gradient variance but also doubles per-step cost. You can scale LR ∝ √B (or ∝ B with linear scaling rule).

Q: What is the linear scaling rule for batch size?
A: \[\eta \leftarrow \eta \cdot \frac{B'}{B} \text{ when changing batch size from } B \text{ to } B'\]
Symbols: \(B' / B\) = batch size multiplier; learning rate scales linearly.
Intuition: To maintain the same effective noise level when multiplying batch size by k, multiply learning rate by k. Used in large-batch training (Goyal et al. 2017 — train ImageNet in 1 hour).

Q: What is the exponential moving average (EMA) of model weights?
A: \[\bar{\theta}_t = \beta\bar{\theta}_{t-1} + (1-\beta)\theta_t\]
Symbols: \(\bar{\theta}\) = EMA weights; \(\beta \approx 0.999\); used for inference.
Intuition: Maintains a smoothed average of model weights over training. EMA models often generalize better than the last checkpoint. Used in BYOL, Stable Diffusion, and many modern training recipes.

Q: What is distributed data-parallel training and its gradient averaging?
A: \[\nabla\mathcal{L} = \frac{1}{K}\sum_{k=1}^K \nabla\mathcal{L}_k\]
Symbols: \(K\) = number of GPUs; \(\nabla\mathcal{L}_k\) = gradient on k-th device's mini-batch.
Intuition: Each GPU computes gradients on its own data, then gradients are averaged (all-reduce) before the parameter update. Equivalent to training with batch size K×B. Linear scaling rule applies.

Q: What is gradient accumulation?
A: \[\theta \leftarrow \theta - \eta \frac{1}{A}\sum_{a=1}^A \nabla\mathcal{L}_{\mathcal{B}_a}\]
Symbols: \(A\) = accumulation steps; \(\mathcal{B}_a\) = mini-batch a; equivalent to batch size \(A \times B\).
Intuition: Simulates large batch training without large GPU memory. Accumulate gradients over A small batches before doing the optimizer step. Common technique when GPU memory limits batch size.

Q: What is the Polyak averaging (iterate averaging)?
A: \[\bar{\theta}_T = \frac{1}{T}\sum_{t=1}^T \theta_t\]
Symbols: \(\bar{\theta}_T\) = average of all iterates up to time T.
Intuition: Averaging SGD iterates provably achieves optimal convergence rates without requiring diminishing step sizes. The averaged model often generalizes better than the final iterate. Related to EMA of weights.

Q: What is the proximal gradient method (ISTA)?
A: \[\theta_{t+1} = \mathrm{prox}_{\eta g}\!\left(\theta_t - \eta\nabla f(\theta_t)\right)\]
Symbols: \(f\) = smooth part; \(g\) = non-smooth part (e.g., L1); \(\mathrm{prox}\) = proximal operator.
Intuition: Handles objectives with both smooth (gradient-based) and non-smooth (proximal) components. ISTA iterates gradient descent step + proximal step. FISTA adds Nesterov acceleration.

Q: What is the mirror descent algorithm?
A: \[\theta_{t+1} = \arg\min_\theta \langle\nabla f(\theta_t), \theta\rangle + \frac{1}{\eta}D_\phi(\theta,\theta_t)\]
Symbols: \(D_\phi\) = Bregman divergence; generalizes gradient descent to non-Euclidean geometry.
Intuition: Instead of Euclidean distance (GD), uses a Bregman divergence to measure 'closeness'. For the negative entropy mirror map → exponentiated gradient. Useful when θ lies on a simplex.

Q: What is the Frank-Wolfe (conditional gradient) method?
A: \[\mathbf{s}_t = \arg\min_{\mathbf{s} \in \mathcal{C}} \langle\nabla f(\theta_t), \mathbf{s}\rangle, \quad \theta_{t+1} = (1-\gamma_t)\theta_t + \gamma_t\mathbf{s}_t\]
Symbols: \(\mathcal{C}\) = convex constraint set; \(\gamma_t\) = step size; projection-free.
Intuition: Linearize objective and minimize over feasible set (linear minimization oracle). No projection needed. Useful for problems where projection is expensive but linear minimization is cheap (e.g., nuclear norm ball).

Q: What is variance reduction in stochastic optimization (SVRG)?
A: \[g_t = \nabla f_{i_t}(\theta_t) - \nabla f_{i_t}(\tilde\theta) + \nabla f(\tilde\theta)\]
Symbols: \(\tilde\theta\) = snapshot of parameters; \(\nabla f(\tilde\theta)\) = full gradient at snapshot.
Intuition: Corrects mini-batch gradient with the full gradient at a periodic snapshot. Dramatically reduces gradient variance → converges like full GD but cost per step close to SGD. Achieves linear convergence for strongly convex problems.

Q: What is the condition for convergence of gradient descent on a strongly convex L-smooth function?
A: \[\|\theta_t - \theta^*\|^2 \leq \left(1 - \frac{m}{L}\right)^t \|\theta_0 - \theta^*\|^2\]
Symbols: \(m\) = strong convexity constant; \(L\) = smoothness; \(\kappa = L/m\) = condition number.
Intuition: Linear (geometric) convergence: each step reduces the error by factor (1-m/L). Well-conditioned problems (κ≈1): fast convergence. Ill-conditioned (κ≫1): slow. Preconditioning reduces κ.

Q: What is the regret in online learning?
A: \[R_T = \sum_{t=1}^T \ell_t(\theta_t) - \min_\theta \sum_{t=1}^T \ell_t(\theta)\]
Symbols: \(\ell_t\) = loss at round t; \(\theta_t\) = online prediction; regret = extra loss vs. best fixed solution.
Intuition: In online learning, you make predictions before seeing the loss function. Regret measures how much worse you did than the best fixed predictor in hindsight. OGD (online gradient descent) achieves O(√T) regret.

Q: What is AdaGrad's regret bound?
A: \[R_T \leq \frac{\|\theta^*\|_2^2}{2\eta}\sum_{j=1}^d\sqrt{\sum_{t=1}^T g_{t,j}^2}\]
Symbols: \(g_{t,j}\) = gradient component j at time t; sum of per-dimension cumulative gradient norms.
Intuition: AdaGrad was originally derived from online learning regret analysis. It's optimal for sparse gradient settings because dimensions with small cumulative gradients get large learning rates.

Q: What is trust region optimization?
A: \[\min_{\delta}\, \nabla f^\top\delta + \frac{1}{2}\delta^\top H\delta \quad \text{s.t.}\quad \|\delta\| \leq \Delta\]
Symbols: \(\Delta\) = trust region radius; \(H\) = Hessian (or approximation).
Intuition: Uses a quadratic model within a trust region where the approximation is reliable. Adapts the step size based on how well the model predicted the actual decrease. Used in TRPO for RL policy optimization.

Q: What is the gradient flow ODE corresponding to gradient descent?
A: \[\frac{d\theta}{dt} = -\nabla\mathcal{L}(\theta)\]
Symbols: \(t\) = continuous time; \(\theta(t)\) = parameter trajectory.
Intuition: As step size → 0, gradient descent becomes a continuous ODE — gradient flow. Analyzing this ODE reveals implicit biases of optimization (e.g., GD on linear networks finds min-norm solutions).

Q: What is coordinate descent?
A: \[\theta_j^{t+1} = \arg\min_{\theta_j} \mathcal{L}(\theta_1^{t+1},\ldots,\theta_{j-1}^{t+1},\theta_j,\theta_{j+1}^t,\ldots)\]
Symbols: Minimize over one coordinate \(\theta_j\) at a time, holding others fixed.
Intuition: Each step optimizes a 1D subproblem. Useful when each coordinate has a closed-form solution (e.g., LASSO via coordinate descent). Efficient for large sparse problems.

Q: What is the subgradient method for non-smooth optimization?
A: \[\theta_{t+1} = \theta_t - \eta_t \mathbf{g}_t, \quad \mathbf{g}_t \in \partial\mathcal{L}(\theta_t)\]
Symbols: \(\partial\mathcal{L}\) = subdifferential; any subgradient can be used.
Intuition: Extends gradient descent to non-smooth objectives (L1 regularization, hinge loss). Does NOT guarantee descent at every step. Requires diminishing step sizes (Robbins-Monro) for convergence.

Q: What is stochastic gradient Langevin dynamics (SGLD)?
A: \[\theta_{t+1} = \theta_t - \eta\nabla\mathcal{L}(\theta_t) + \sqrt{2\eta}\,\boldsymbol{\epsilon}_t, \quad \boldsymbol{\epsilon}_t \sim \mathcal{N}(0,I)\]
Symbols: \(\boldsymbol{\epsilon}_t\) = Gaussian noise; turns SGD into a sampler.
Intuition: Adds Langevin noise to SGD — bridges optimization and Bayesian sampling. For small η, samples from the posterior ∝ exp(-L(θ)). Used in Bayesian deep learning.

Q: What is the natural gradient and why does it differ from the standard gradient?
A: \[\tilde\nabla_\theta \mathcal{L} = F(\theta)^{-1}\nabla_\theta \mathcal{L}, \quad F = \mathcal{I}(\theta)\]
Symbols: \(F\) = Fisher information matrix; \(\tilde\nabla\) = natural gradient.
Intuition: Standard gradient is defined in parameter space (Euclidean); natural gradient is in distribution space (Riemannian). Takes equal steps in distribution space, invariant to parameterization. Used in K-FAC and TRPO.

Q: What is K-FAC (Kronecker-Factored Approximate Curvature)?
A: \[F_l \approx A_{l-1} \otimes G_l, \quad A_{l-1} = \mathbb{E}[aa^\top],\; G_l = \mathbb{E}[gg^\top]\]
Symbols: \(A_{l-1}\) = activation covariance; \(G_l\) = gradient covariance; \(\otimes\) = Kronecker product.
Intuition: Approximates the Fisher information matrix as a Kronecker product of two smaller matrices per layer. Makes natural gradient tractable for large networks. Used to train networks faster with better conditioning.

Q: What is the Armijo line search condition?
A: \[\mathcal{L}(\theta - \eta d) \leq \mathcal{L}(\theta) - c\,\eta\,\nabla\mathcal{L}^\top d\]
Symbols: \(d\) = descent direction; \(c \in (0,1)\) = sufficient decrease parameter; ensures adequate reduction.
Intuition: A sufficient decrease condition — ensures each step reduces the loss by at least a fraction c of what the linear approximation predicts. Used in backtracking line search to choose step size adaptively.

Q: What is the Wolfe condition for line search?
A: \[\text{Armijo + curvature condition: } \nabla\mathcal{L}(\theta-\eta d)^\top d \geq c_2 \nabla\mathcal{L}^\top d\]
Symbols: \(c_2 \in (c_1,1)\); curvature condition prevents steps too short.
Intuition: Wolfe conditions (Armijo + curvature) ensure the line search finds a point that sufficiently decreases both the objective and the gradient. Standard in quasi-Newton methods like L-BFGS.

Q: What is the conjugate gradient method?
A: \[d_{t+1} = -\nabla\mathcal{L}_{t+1} + \beta_t d_t, \quad \beta_t = \frac{\|\nabla\mathcal{L}_{t+1}\|^2}{\|\nabla\mathcal{L}_t\|^2}\]
Symbols: \(d_t\) = conjugate direction; \(\beta_t\) = Polak-Ribière formula; search directions are A-conjugate.
Intuition: Builds on momentum — each search direction is conjugate to all previous ones, so each step doesn't 'undo' previous progress. Converges in n steps for quadratic objectives. Used for large-scale linear systems.

Q: What is the Polyak step size in subgradient methods?
A: \[\eta_t = \frac{\mathcal{L}(\theta_t) - \mathcal{L}^*}{\|\mathbf{g}_t\|^2}\]
Symbols: \(\mathcal{L}^*\) = optimal loss value (must be known or estimated); \(\mathbf{g}_t\) = subgradient.
Intuition: Optimal step size for subgradient methods when the optimal value is known. Adapts step size based on distance to optimal. In practice, replace L* with current best lower bound.

Q: What is the role of epsilon in Adam to prevent division by zero?
A: \[\theta_{t+1} = \theta_t - \frac{\eta\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}, \quad \epsilon \approx 10^{-7} \text{ to } 10^{-8}\]
Symbols: \(\epsilon\) = numerical stability constant; prevents division by zero when \(\hat{v}_t \approx 0\).
Intuition: When a parameter receives near-zero gradients, v̂ ≈ 0 and division would be numerically unstable. ε adds a small floor. Larger ε → more similar to SGD; smaller ε → more adaptive behavior.

Q: What is gradient noise in the context of large language model training?
A: \[\sigma_g^2 = \frac{S}{\text{batch size}}, \quad S = \frac{1}{N}\sum_i \|\nabla\ell_i - \nabla\mathcal{L}\|^2\]
Symbols: \(S\) = gradient noise scale; \(N\) = dataset size; \(\nabla\ell_i\) = per-sample gradient.
Intuition: Gradient noise scale S measures the signal-to-noise ratio of mini-batch gradients. OpenAI found that critical batch size ≈ S determines when batch size stops helping — theory for LLM training budget allocation.

Q: What is the LAMB optimizer?
A: \[\theta_j \leftarrow \theta_j - \eta \frac{\|\theta_j\|}{\|\hat{m}_j/\sqrt{\hat{v}_j}\|}\cdot\frac{\hat{m}_j}{\sqrt{\hat{v}_j}+\epsilon}\]
Symbols: \(\|\theta_j\|/\|\hat{m}_j/\sqrt{\hat{v}_j}\|\) = per-layer trust ratio; scales Adam update by layer norm.
Intuition: LAMB (Layer-wise Adaptive Moments) adapts Adam for large-batch distributed training. Per-layer trust ratio prevents exploding or vanishing updates in different layers. Enables training BERT in 76 minutes.

Q: What is mixed precision training and gradient scaling?
A: \[s \cdot \mathcal{L} \to \text{backward} \to s^{-1}\mathbf{g}\]
Symbols: \(s\) = loss scale (e.g., 1024); prevents underflow in FP16 gradients; \(s^{-1}\mathbf{g}\) = unscaled gradient.
Intuition: FP16 has limited dynamic range. Scale the loss up before backward pass to keep gradients in FP16 range, then scale down before optimizer step. NVIDIA Apex and PyTorch AMP automate this.

Q: What is gradient checkpointing and its memory-computation tradeoff?
A: \[\text{Memory: } O(\sqrt{N}) \text{ vs. } O(N); \quad \text{Compute: } \approx 33\% \text{ overhead}\]
Symbols: \(N\) = number of layers; recomputes activations during backward pass instead of storing.
Intuition: Normally, all activations are stored for backward pass (O(N) memory). Checkpointing stores only √N checkpoints and recomputes the rest. Trades 33% extra computation for √N memory savings — enables training larger models.

Q: What is the difference between offline and online learning in optimization?
A: \[\text{Offline: } \min_\theta \sum_{i=1}^N \ell_i(\theta);\quad \text{Online: sequential } \min \text{ regret}\]
Symbols: Offline has all data upfront; online processes data one at a time adaptively.
Intuition: Offline (standard ML): whole dataset available, minimize empirical loss. Online: data arrives sequentially; goal is low regret against best offline predictor. Online algorithms include AdaGrad, which was derived in this setting.

Q: What is the mathematical model of a single neuron?
A: \[a = \sigma(z), \quad z = \mathbf{w}^\top\mathbf{x} + b\]
Symbols: \(\mathbf{w}\) = weight vector; \(b\) = bias; \(\sigma\) = activation function; \(z\) = pre-activation; \(a\) = activation output.
Intuition: Computes a weighted sum of inputs plus a bias (linear part), then applies a non-linear activation. The weights and bias are learned. Without σ, stacking layers is equivalent to one linear layer.

Q: What is the matrix form of a neural network layer?
A: \[\mathbf{z} = W\mathbf{x} + \mathbf{b}, \quad \mathbf{a} = \sigma(\mathbf{z})\]
Symbols: \(W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}\); \(\mathbf{b} \in \mathbb{R}^{d_{\text{out}}}\); activation applied element-wise.
Intuition: A layer is a linear transformation (W, b) followed by a non-linear activation. All N samples can be processed as a batch: Z = XW^T + b with X ∈ ℝ^{N×d_in}.

Q: What is network depth vs width?
A: \[\text{Depth} = L \text{ (number of layers)}; \quad \text{Width} = d_l \text{ (units per layer)}\]
Symbols: \(L\) = number of layers; \(d_l\) = hidden units in layer l.
Intuition: Depth allows hierarchical feature extraction (each layer builds on the last). Width controls representational capacity per layer. Deep networks can represent exponentially more functions than shallow wide ones.

Q: What is the sigmoid activation function and its derivative?
A: \[\sigma(x) = \frac{1}{1+e^{-x}},\quad \sigma'(x) = \sigma(x)(1-\sigma(x))\]
Symbols: Output \(\in (0,1)\); max derivative = 0.25 at x=0.
Intuition: Squashes inputs to (0,1). Used for binary output layers. Saturates at ±∞ (gradient ≈ 0) causing vanishing gradients in deep networks — replaced by ReLU in hidden layers.

Q: What is the tanh activation function and its derivative?
A: \[\tanh(x) = \frac{e^x-e^{-x}}{e^x+e^{-x}},\quad \tanh'(x) = 1-\tanh^2(x)\]
Symbols: Output \(\in (-1,1)\); zero-centered unlike sigmoid.
Intuition: Similar to sigmoid but zero-centered (mean output ≈ 0) which helps gradient flow. Still saturates for large |x|. Preferred over sigmoid for hidden layers; still used in LSTMs.

Q: What is the ReLU activation function?
A: \[\text{ReLU}(x) = \max(0,x) = \begin{cases}x & x>0 \\ 0 & x\leq 0\end{cases}\]
Symbols: Rectified linear unit; derivative = 1 for x>0, 0 for x<0.
Intuition: Simple and effective: no saturation for positive inputs (gradient = 1, no vanishing). Fast to compute. Problem: dying ReLU — if z is always ≤0, neuron gets no gradient and 'dies'. Default choice for hidden layers.

Q: What is the dying ReLU problem?
A: \[x < 0 \implies \frac{\partial \text{ReLU}(x)}{\partial x} = 0 \implies \frac{\partial \mathcal{L}}{\partial w} = 0\]
Symbols: If pre-activation is always negative, gradient is always zero — no learning.
Intuition: A neuron 'dies' if its pre-activation stays negative for all training samples. Caused by too-large learning rates or unfortunate initialization. Leaky ReLU and He initialization mitigate this.

Q: What is Leaky ReLU?
A: \[\text{LeakyReLU}(x) = \begin{cases}x & x>0 \\ \alpha x & x\leq 0\end{cases}, \quad \alpha \approx 0.01\]
Symbols: \(\alpha\) = small negative slope (fixed or learned — PReLU).
Intuition: Allows a small gradient for negative inputs, preventing neuron death. PReLU (parametric) learns α. ELU uses exp for negative inputs — smoother but more expensive.

Q: What is the GELU activation function?
A: \[\text{GELU}(x) = x\,\Phi(x) = x \cdot \frac{1}{2}\left[1+\mathrm{erf}\!\left(\frac{x}{\sqrt{2}}\right)\right]\]
Symbols: \(\Phi(x)\) = standard normal CDF; \(\mathrm{erf}\) = error function.
Intuition: Weights input by its probability of being greater than 0 under a standard Gaussian. Smooth everywhere (unlike ReLU). Widely used in transformers (BERT, GPT). Can be approximated as 0.5x(1+tanh(√(2/π)(x+0.044715x³))).

Q: What is the softmax activation function?
A: \[\text{softmax}(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^K e^{z_j}}\]
Symbols: \(\mathbf{z} \in \mathbb{R}^K\) = logits; output is a probability distribution over K classes; \(\sum_i \text{softmax}(\mathbf{z})_i = 1\).
Intuition: Converts logits to probabilities. Differentiable approximation of argmax. Temperature-scaled: softmax(z/T) → uniform as T→∞, one-hot as T→0.

Q: What is the numerically stable computation of softmax?
A: \[\text{softmax}(\mathbf{z})_i = \frac{e^{z_i - \max_j z_j}}{\sum_j e^{z_j - \max_j z_j}}\]
Symbols: Subtracting \(\max_j z_j\) doesn't change the result but prevents overflow.
Intuition: Without shifting, large logits cause e^z to overflow. Shifting by max(z) makes the largest term 1 (e^0=1) and smaller terms <1. Identical result mathematically, stable numerically.

Q: What is softmax temperature scaling?
A: \[\text{softmax}(\mathbf{z}/T)_i = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}}\]
Symbols: \(T\) = temperature; \(T<1\) sharpens; \(T>1\) flattens distribution.
Intuition: Low T → near one-hot (model more confident/decisive). High T → near uniform (more exploratory/uncertain). Used in knowledge distillation, sampling from LLMs, and reinforcement learning.

Q: What is the Mean Squared Error (MSE) loss?
A: \[\mathcal{L}_{\mathrm{MSE}} = \frac{1}{N}\sum_{i=1}^N (y_i - \hat{y}_i)^2\]
Symbols: \(y_i\) = true target; \(\hat{y}_i\) = prediction; squared penalty for each error.
Intuition: Standard regression loss. Penalizes large errors more heavily (squared). MLE for Gaussian noise model. Gradient: 2(ŷ-y)/N. Sensitive to outliers due to squaring.

Q: What is the Mean Absolute Error (MAE) loss?
A: \[\mathcal{L}_{\mathrm{MAE}} = \frac{1}{N}\sum_{i=1}^N |y_i - \hat{y}_i|\]
Symbols: Absolute value instead of square; gradient = sign(ŷ-y).
Intuition: More robust to outliers than MSE (linear not quadratic penalty). But non-differentiable at 0, and gradient doesn't decrease near minimum. Huber loss (smooth L1) combines best of both.

Q: What is the Huber loss?
A: \[\mathcal{L}_\delta(r) = \begin{cases}\frac{1}{2}r^2 & |r| \leq \delta \\ \delta|r| - \frac{1}{2}\delta^2 & |r| > \delta\end{cases}\]
Symbols: \(r = y - \hat{y}\) = residual; \(\delta\) = threshold between quadratic and linear regions.
Intuition: Quadratic for small residuals (smooth, fast convergence), linear for large residuals (outlier-robust). Best of MSE and MAE. Used in regression with outliers (DQN Q-value loss).

Q: What is the hinge loss?
A: \[\mathcal{L}_{\mathrm{hinge}} = \max(0, 1 - y\hat{f})\]
Symbols: \(y \in \{-1,+1\}\) = binary label; \(\hat{f}\) = model output (decision value).
Intuition: SVM loss. Penalizes predictions with wrong sign or insufficient margin (|ŷ|<1). Zero loss for correct confident predictions (ŷ·y ≥ 1). Non-differentiable at margin boundary (subgradient = 0 or -y).

Q: What is the backpropagation algorithm?
A: \[\delta^{(l)} = \left((W^{(l+1)})^\top \delta^{(l+1)}\right) \odot \sigma'(z^{(l)})\]
Symbols: \(\delta^{(l)}\) = error signal at layer l; propagated backward by transposed weight matrix and elementwise activation derivative.
Intuition: Efficient algorithm to compute ∂L/∂W for all layers simultaneously using the chain rule. Backward pass mirrors the forward pass in reverse, reusing activations computed during forward.

Q: What is the gradient of the loss w.r.t. weights in backpropagation?
A: \[\frac{\partial \mathcal{L}}{\partial W^{(l)}} = \delta^{(l)} (a^{(l-1)})^\top\]
Symbols: \(\delta^{(l)}\) = error at layer l; \(a^{(l-1)}\) = activation from previous layer.
Intuition: Outer product of error signal and previous layer activation. Weight gradient = upstream gradient times pre-synaptic activation. This is why large activations amplify weight gradients.

Q: What is the gradient of the loss w.r.t. biases in backpropagation?
A: \[\frac{\partial \mathcal{L}}{\partial \mathbf{b}^{(l)}} = \delta^{(l)}\]
Symbols: \(\delta^{(l)}\) = error signal at layer l; bias gradient = error signal directly.
Intuition: Bias gradient equals the error signal — because ∂z/∂b = 1 (bias contributes additively with coefficient 1).

Q: What is the gradient of the MSE loss at the output layer?
A: \[\delta^{(L)} = \hat{\mathbf{y}} - \mathbf{y}\]
Symbols: \(\hat{\mathbf{y}}\) = prediction; \(\mathbf{y}\) = target; for identity output activation.
Intuition: The error signal at the output is simply the residual (prediction minus target). Clean formula makes regression training straightforward.

Q: What is the gradient of cross-entropy + softmax at the output layer?
A: \[\delta^{(L)} = \hat{\mathbf{y}} - \mathbf{y}\]
Symbols: \(\hat{\mathbf{y}}\) = softmax probabilities; \(\mathbf{y}\) = one-hot label.
Intuition: Cross-entropy loss combined with softmax output gives a beautifully simple gradient: just prediction minus label. This is why softmax + cross-entropy is the standard for classification.

Q: What is the vanishing gradient problem?
A: \[\frac{\partial \mathcal{L}}{\partial W^{(1)}} = \prod_{l=1}^{L-1}\left(W^{(l)}\right)^\top \cdot \delta^{(L)}\]
Symbols: Product of L-1 Jacobians; if each factor < 1, product → 0 exponentially.
Intuition: In deep networks, gradient must pass through many layers. If activation derivatives and weight magnitudes are <1, the product decays exponentially — early layers barely learn. Solved by ReLU, batch norm, residual connections.

Q: What is the exploding gradient problem?
A: \[\left\|\prod_{l=1}^{L}\frac{\partial \mathbf{h}^{(l)}}{\partial \mathbf{h}^{(l-1)}}\right\| > 1^L \implies \|\text{gradient}\| \to \infty\]
Symbols: Product of Jacobians with norm > 1 grows exponentially with depth.
Intuition: When weight matrices have spectral norm >1, gradient products explode. Common in RNNs (through time). Solution: gradient clipping, careful initialization, batch normalization.

Q: What is Xavier (Glorot) initialization?
A: \[W \sim \mathcal{N}\!\left(0,\, \frac{2}{n_{\text{in}}+n_{\text{out}}}\right)\]
Symbols: \(n_{\text{in}}, n_{\text{out}}\) = number of input and output units; variance chosen to preserve signal variance.
Intuition: Designed for tanh/sigmoid activations. Sets variance to keep gradient magnitudes roughly constant across layers — prevents vanishing/exploding. Derived by requiring Var(output) = Var(input) in forward pass.

Q: What is He (Kaiming) initialization?
A: \[W \sim \mathcal{N}\!\left(0,\, \frac{2}{n_{\text{in}}}\right)\]
Symbols: \(n_{\text{in}}\) = fan-in; factor of 2 accounts for ReLU killing half the outputs (sets E[a²] = Var(z)).
Intuition: Designed for ReLU activations. ReLU sets half the activations to 0, so needs larger initial variance to maintain signal. Default initialization for ReLU networks in PyTorch.

Q: Why does proper initialization matter for deep networks?
A: \[\mathrm{Var}(a^{(l)}) = n_{\text{in}} \cdot \mathrm{Var}(W) \cdot \mathrm{Var}(a^{(l-1)})\]
Symbols: Layer output variance scales with layer width and weight variance.
Intuition: If weight variance is too large, activations explode. Too small, they vanish to zero. Xavier/He set variance to exactly cancel the fan-in scaling, maintaining unit variance throughout the network.

Q: What is batch normalization?
A: \[\hat{x}_i = \frac{x_i - \mu_\mathcal{B}}{\sqrt{\sigma_\mathcal{B}^2+\epsilon}}, \quad y_i = \gamma\hat{x}_i + \beta\]
Symbols: \(\mu_\mathcal{B}, \sigma_\mathcal{B}^2\) = batch mean and variance; \(\gamma, \beta\) = learned scale and shift; \(\epsilon\) = stability.
Intuition: Normalizes each mini-batch to zero mean, unit variance, then rescales with learned γ,β. Reduces internal covariate shift — allows higher learning rates and is less sensitive to initialization.

Q: What is the batch normalization forward pass during inference?
A: \[\hat{x} = \frac{x - \mu_{\mathrm{running}}}{\sqrt{\sigma_{\mathrm{running}}^2+\epsilon}}, \quad \mu_{\mathrm{running}} = (1-m)\mu_{\mathrm{running}} + m\mu_\mathcal{B}\]
Symbols: \(\mu_{\mathrm{running}}\) = exponential moving average tracked during training; \(m\) = momentum.
Intuition: At inference, use running statistics (no mini-batch available). These are accumulated during training via exponential moving average. Makes inference independent of batch composition.

Q: What is layer normalization?
A: \[\hat{x}_i = \frac{x_i - \mu_x}{\sqrt{\sigma_x^2+\epsilon}}, \quad \mu_x = \frac{1}{d}\sum_j x_j\]
Symbols: Normalize across the feature dimension (not batch); \(d\) = feature dimension.
Intuition: Normalizes across features within a single example (not across batch). Works with any batch size including 1. Preferred in transformers and RNNs where batch statistics are unreliable. Followed by learned γ,β.

Q: What is the difference between batch normalization and layer normalization?
A: \[\text{BN: normalize over batch (axis 0); LN: normalize over features (axis 1)}\]
Symbols: BN statistics: \(\mu_{\mathcal{B}} = \frac{1}{B}\sum_{n=1}^B x_n\); LN statistics: \(\mu_x = \frac{1}{d}\sum_{j=1}^d x_{n,j}\).
Intuition: BN depends on the batch → requires large batch for stable statistics and running means for inference. LN is per-sample → batch-independent. Transformers use LN (variable sequence lengths, small batches).

Q: What is L2 regularization (weight decay) added to the loss?
A: \[\mathcal{L}_{\mathrm{reg}} = \mathcal{L} + \frac{\lambda}{2}\sum_j w_j^2\]
Symbols: \(\lambda\) = regularization strength; \(w_j\) = weights (not biases typically).
Intuition: Penalizes large weights — encourages smooth, simple models. Gradient: ∇L_reg = ∇L + λw. Equivalent to Gaussian prior on weights in MAP estimation.

Q: What is L1 regularization added to the loss?
A: \[\mathcal{L}_{\mathrm{reg}} = \mathcal{L} + \lambda\sum_j |w_j|\]
Symbols: \(\lambda\) = regularization strength; L1 subgradient: \(\mathrm{sign}(w_j)\).
Intuition: Penalizes the sum of absolute weight values. Produces sparse models — many weights driven exactly to zero (unlike L2 which only shrinks them). Equivalent to Laplace prior. Useful for feature selection.

Q: What is dropout regularization?
A: \[\tilde{a}_i = \frac{a_i \cdot m_i}{1-p}, \quad m_i \sim \mathrm{Bernoulli}(1-p)\]
Symbols: \(m_i\) = binary mask; \(p\) = dropout probability; \(1/(1-p)\) = inverted dropout scaling.
Intuition: Randomly zeroes neurons during training (prevents co-adaptation). Scaling by 1/(1-p) ensures expected output unchanged at test time. Acts as ensemble training — each forward pass uses a different sub-network.

Q: What is the expected output of a dropout layer at test time?
A: \[\mathbb{E}[\tilde{a}] = \mathbb{E}\!\left[\frac{a \cdot m}{1-p}\right] = a\]
Symbols: \(\mathbb{E}[m] = 1-p\); \(1/(1-p)\) correction makes \(\mathbb{E}[\tilde{a}] = a\).
Intuition: Inverted dropout ensures the expected activation at test time equals the training activation (no rescaling needed at test time). Without this, you'd need to multiply all weights by (1-p) at test time.

Q: What is the universal approximation theorem?
A: \[\forall\, f \in C(\mathbb{R}^n,\mathbb{R}),\, \epsilon>0:\; \exists \text{ NN with 1 hidden layer s.t. } \sup|\hat{f}-f|<\epsilon\]
Symbols: \(C(\mathbb{R}^n,\mathbb{R})\) = continuous functions; width may need to be exponentially large.
Intuition: A single hidden layer with enough neurons can approximate any continuous function to arbitrary precision. Doesn't say how many neurons or how to find them — provides existence, not efficiency.

Q: Does the universal approximation theorem guarantee good learning in practice?
A: \[\text{Existence } \neq \text{ learnability by SGD on finite data}\]
Symbols: UAT: there exists a NN that fits; SGD: finds one via optimization on N samples.
Intuition: UAT only guarantees existence of a network with the right weights — not that gradient descent will find it, or that it will generalize. Depth helps: deep networks require exponentially fewer neurons than wide ones.

Q: What is a 1D discrete convolution?
A: \[(f * g)[n] = \sum_{m=-\infty}^{\infty} f[m]\,g[n-m]\]
Symbols: \(f\) = input; \(g\) = filter/kernel; \(n\) = output position.
Intuition: Slides the filter over the input, computing dot products at each position. Translation-equivariant: if input shifts, output shifts by the same amount. Parameter sharing: same filter weights applied everywhere.

Q: What is a 2D convolution in CNNs?
A: \[(I * K)[i,j] = \sum_{m}\sum_{n} I[i+m,\,j+n]\,K[m,n]\]
Symbols: \(I\) = input feature map; \(K\) = kernel; \((i,j)\) = output spatial position.
Intuition: Slides a 2D filter over a 2D input. Each output pixel = weighted sum of a local patch. Strong inductive bias: local connectivity + translation equivariance. Huge parameter efficiency vs. fully connected.

Q: What are pooling operations in CNNs?
A: \[\text{MaxPool}(X)_{ij} = \max_{(m,n) \in \text{window}} X_{im,jn}; \quad \text{AvgPool} = \text{mean}\]
Symbols: Reduces spatial dimensions; window = pooling kernel size.
Intuition: Downsamples feature maps to reduce spatial resolution and parameter count. Max pooling selects the most active feature in each region (introduces some translation invariance). Average pooling computes the mean.

Q: What is a residual (skip) connection?
A: \[\mathbf{y} = F(\mathbf{x}, \{W_i\}) + \mathbf{x}\]
Symbols: \(F(\mathbf{x})\) = residual mapping learned by the sub-block; \(\mathbf{x}\) = identity shortcut.
Intuition: The block only needs to learn the residual (difference from input). If the optimal output ≈ input, residual → 0 (easy to learn). Gradient flows directly through the skip connection, solving vanishing gradient in very deep networks.

Q: Why do residual connections help gradient flow?
A: \[\frac{\partial \mathcal{L}}{\partial \mathbf{x}} = \frac{\partial \mathcal{L}}{\partial \mathbf{y}}\frac{\partial \mathbf{y}}{\partial \mathbf{x}} = \frac{\partial \mathcal{L}}{\partial \mathbf{y}}\left(1 + \frac{\partial F}{\partial \mathbf{x}}\right)\]
Symbols: The '1' term = direct gradient path through skip connection; never vanishes.
Intuition: Even if ∂F/∂x → 0 (deep layers not contributing), gradient still flows through the identity shortcut. This guarantees non-vanishing gradients regardless of depth.

Q: What is the ELU (Exponential Linear Unit) activation?
A: \[\text{ELU}(x) = \begin{cases}x & x>0 \\ \alpha(e^x-1) & x\leq 0\end{cases}\]
Symbols: \(\alpha > 0\) = saturation value for negative inputs (often α=1); smooth at x=0.
Intuition: Saturates to -α for large negative inputs but has non-zero gradient for x≤0 (no dying neuron). Smooth everywhere, mean activations closer to zero. Better than ReLU in practice for some tasks.

Q: What is the Swish activation function?
A: \[\text{Swish}(x) = x \cdot \sigma(x) = \frac{x}{1+e^{-x}}\]
Symbols: \(\sigma(x)\) = sigmoid; self-gated activation; smooth and non-monotonic.
Intuition: Like ReLU but smooth and slightly negative for small x. Empirically outperforms ReLU on deep networks. GELU ≈ Swish with β≈1.702. Used in EfficientNet and some transformer variants.

Q: What is the SiLU activation (Sigmoid Linear Unit)?
A: \[\text{SiLU}(x) = x\sigma(x)\]
Symbols: SiLU = Swish with β=1; smooth, non-monotonic, unbounded above.
Intuition: Equivalent to Swish(x, β=1). Used in LLaMA and many modern LLMs as a drop-in for GELU. Combines linear and sigmoidal properties for better gradient flow.

Q: What is the GLU (Gated Linear Unit)?
A: \[\text{GLU}(\mathbf{x}) = \mathbf{x}_1 \odot \sigma(\mathbf{x}_2), \quad \mathbf{x} = [\mathbf{x}_1;\mathbf{x}_2]\]
Symbols: \(\mathbf{x}_1, \mathbf{x}_2\) = two halves of the linear projection; \(\odot\) = element-wise product.
Intuition: One half gates the other. SwiGLU = Swish(x₁)⊙x₂. Used in Llama's FFN layers instead of standard ReLU/GELU FFN for better performance.

Q: What is the forward pass of a multi-layer network (depth L)?
A: \[\mathbf{a}^{(0)} = \mathbf{x}, \quad \mathbf{z}^{(l)} = W^{(l)}\mathbf{a}^{(l-1)}+\mathbf{b}^{(l)}, \quad \mathbf{a}^{(l)} = \sigma(\mathbf{z}^{(l)})\]
Symbols: \(l = 1,\ldots,L\); \(\mathbf{a}^{(L)} = \hat{\mathbf{y}}\) = output.
Intuition: Each layer transforms its input via a linear map then a non-linearity. Data flows from input to output, building increasingly abstract representations.

Q: What is the Jacobian of a softmax layer?
A: \[J_{ij} = \frac{\partial s_i}{\partial z_j} = s_i(\delta_{ij} - s_j)\]
Symbols: \(s_i\) = softmax output i; \(\delta_{ij}\) = Kronecker delta; \(J \in \mathbb{R}^{K\times K}\).
Intuition: Diagonal (i=j): s_i(1-s_i) = s_i × (1 - itself). Off-diagonal (i≠j): -s_i·s_j. In practice combined with cross-entropy: gradient simplifies to s-y.

Q: What is the computational cost of backpropagation vs forward pass?
A: \[\text{Backward cost} \approx 2 \times \text{Forward cost}\]
Symbols: Rough rule of thumb; actual ratio depends on architecture.
Intuition: Backprop requires the same amount of computation as two forward passes (one to compute activations, one to propagate gradients). Total training step ≈ 3× forward cost. Memory ∝ depth (must store all activations).

Q: What is the sparse softmax (sparsemax) activation?
A: \[\mathrm{sparsemax}(\mathbf{z}) = \arg\min_{\mathbf{p}\in\Delta^{K-1}} \|\mathbf{p}-\mathbf{z}\|^2\]
Symbols: \(\Delta^{K-1}\) = probability simplex; projects z onto the simplex.
Intuition: Alternative to softmax that can output exactly zero probabilities (true sparsity). Useful when you want the model to focus on a few options. Gradient is the identity on the support, zero elsewhere.

Q: What is the attention masking operation in the forward pass?
A: \[A_{ij} = \begin{cases}-\infty & j > i \\ z_{ij} & j \leq i\end{cases}\]
Symbols: \(z_{ij}\) = raw attention score; \(-\infty\) → softmax outputs 0 (masked out).
Intuition: Causal masking prevents position i from attending to future positions j>i. Setting to -∞ before softmax makes those positions receive zero weight after softmax. Essential for autoregressive language models.

Q: What is the input-output relationship for a fully connected network in matrix form?
A: \[\hat{Y} = \sigma^{(L)}\!\left(W^{(L)}\sigma^{(L-1)}\!\left(\cdots\sigma^{(1)}\!\left(W^{(1)}X^\top + \mathbf{b}^{(1)}\right)\right) + \mathbf{b}^{(L)}\right)\]
Symbols: \(X \in \mathbb{R}^{N\times d}\) = batch; \(\hat{Y} \in \mathbb{R}^{N\times K}\) = predictions; L = depth.
Intuition: The complete forward pass stacks L layers of linear + activation. Non-linearities σ are necessary — without them, composition of linear maps is still linear.

Q: What is the neural tangent kernel (NTK)?
A: \[K(\mathbf{x},\mathbf{x}') = \nabla_\theta f_\theta(\mathbf{x})^\top \nabla_\theta f_\theta(\mathbf{x}')\]
Symbols: \(\nabla_\theta f_\theta\) = gradient of network output w.r.t. parameters; inner product in parameter space.
Intuition: In the infinite-width limit, neural networks behave like kernel methods with this kernel. NTK remains constant during training. Explains generalization of overparameterized networks via implicit regularization.

Q: What is weight sharing in neural networks?
A: \[W_1 = W_2 = \cdots = W_T \text{ (same weights across positions/time)}\]
Symbols: Tied weights; greatly reduces parameter count.
Intuition: CNNs share filter weights across spatial positions. RNNs share weights across time steps. Transformers share attention weights across positions. Reduces parameters, enforces translation/temporal equivariance, improves generalization.

Q: What is the parameter count of a fully connected layer?
A: \[|\theta| = d_{\text{in}} \times d_{\text{out}} + d_{\text{out}}\]
Symbols: \(d_{\text{in}} \times d_{\text{out}}\) = weight matrix entries; \(d_{\text{out}}\) = bias terms.
Intuition: Each output unit connects to all input units (weight) plus has a bias. A 768→768 layer has 768²+768 ≈ 590K parameters. Deep networks have millions/billions total.

Q: What is the concept of feature learning in deep networks?
A: \[\mathbf{h}^{(l)} = f^{(l)}\circ f^{(l-1)}\circ \cdots \circ f^{(1)}(\mathbf{x})\]
Symbols: \(\mathbf{h}^{(l)}\) = learned representation at layer l; \(f^{(l)}\) = l-th layer transformation.
Intuition: Each layer transforms the representation — early layers learn low-level features (edges, n-grams), deeper layers learn high-level abstractions (shapes, concepts). The network learns its own features end-to-end.

Q: What is the difference between a shallow and deep network representation?
A: \[|h_{\text{shallow}}| = 2^{d} \text{ vs. } |h_{\text{deep}}| = O(poly(d)) \text{ for some functions}\]
Symbols: \(d\) = input dimension; depth provides exponential parameter savings for structured functions.
Intuition: Some functions require exponentially more neurons to represent in a shallow network than a deep one. Depth = representational efficiency. Real-world data (images, text) has hierarchical structure that depth exploits.

Q: What is the receptive field in a CNN?
A: \[\text{Receptive field at layer } l = k + (k-1)(l-1) \text{ (for kernel size k)}\]
Symbols: \(k\) = kernel size; \(l\) = depth; grows linearly with depth for standard convolution.
Intuition: How large a region of the input influences one output neuron. Deeper layers see a larger context. Dilated convolutions grow the receptive field exponentially (used in WaveNet, DeepLab).

Q: What is the difference between transposed convolution and upsampling?
A: \[\text{TransposedConv: learnable upsampling; Upsample: nearest/bilinear (fixed)}\]
Symbols: TransposedConv = fractionally-strided convolution; increases spatial resolution.
Intuition: Used in decoders and generators (GANs, autoencoders) to increase spatial resolution. Transposed conv learns the upsampling; bilinear interpolation is a fixed mathematical operation. Transposed conv can produce 'checkerboard' artifacts.

Q: What is the SoftPlus function and its relation to ReLU?
A: \[\text{softplus}(x) = \ln(1+e^x), \quad \frac{d}{dx}\text{softplus}(x) = \sigma(x)\]
Symbols: \(\text{softplus}(x) \to x\) for large x (like ReLU); \(\text{softplus}(x) \to 0\) for large negative x.
Intuition: Smooth approximation to ReLU. Its derivative is the sigmoid. Useful when differentiability at 0 matters (e.g., parameterizing positive quantities in probabilistic models).

Q: What is Group Normalization?
A: \[\mu_g = \frac{1}{|G|}\sum_{i \in G} x_i, \quad \hat{x}_i = \frac{x_i-\mu_g}{\sqrt{\sigma_g^2+\epsilon}}\]
Symbols: \(G\) = group of channels; normalizes within groups; independent of batch size.
Intuition: Divides channels into groups, normalizes within each group. Between BN (full batch) and LN (all channels). Works well for small batch sizes (object detection, segmentation). Used in vision transformers.

Q: What is Instance Normalization?
A: \[\hat{x}_{n,c,h,w} = \frac{x_{n,c,h,w} - \mu_{n,c}}{\sqrt{\sigma_{n,c}^2+\epsilon}},\quad \mu_{n,c} = \frac{1}{HW}\sum_{h,w} x_{n,c,h,w}\]
Symbols: Normalizes each sample's each channel independently over spatial dimensions H,W.
Intuition: Per-instance, per-channel normalization. Removes instance-specific contrast information. Used in neural style transfer — normalizing style statistics enables better style transfer.

Q: What is spectral normalization of weight matrices?
A: \[\bar{W} = \frac{W}{\sigma_{\max}(W)}, \quad \sigma_{\max} = \text{largest singular value}\]
Symbols: \(\sigma_{\max}(W)\) = spectral norm; ensures \(\|\bar{W}\|_2 = 1\) (1-Lipschitz layer).
Intuition: Normalizes weights so each layer is 1-Lipschitz. Stabilizes GAN training by preventing the discriminator from varying too rapidly. Computed efficiently via power iteration. Standard in SNGAN.

Q: What is the parameter-efficiency comparison between CNNs and fully connected networks?
A: \[\text{FC: } d_{\text{in}}\times d_{\text{out}}; \quad \text{Conv: } k^2 \times C_{\text{in}} \times C_{\text{out}}\]
Symbols: \(k\) = kernel size; \(C_{\text{in}}, C_{\text{out}}\) = input/output channels; independent of spatial size.
Intuition: A conv layer with k=3, 64 channels in/out has 3²×64×64=36,864 params — vs. a FC layer connecting 64×H×W input to 64×H×W output which has (64HW)² params. Enormous savings via weight sharing.

Q: What is knowledge distillation?
A: \[\mathcal{L}_{\mathrm{KD}} = T^2 D_{\mathrm{KL}}(\text{softmax}(z_T/T)\|\text{softmax}(z_S/T))\]
Symbols: \(T\) = temperature; \(z_T, z_S\) = teacher/student logits; \(T^2\) corrects gradient scale.
Intuition: Train a small (student) network to mimic a large (teacher) network's soft probability outputs. High temperature reveals the teacher's 'dark knowledge' about inter-class similarity. Student learns richer information than from hard labels alone.

Q: What is the LSTM forget gate equation?
A: \[f_t = \sigma(W_f[\mathbf{h}_{t-1};\mathbf{x}_t] + \mathbf{b}_f)\]
Symbols: \(f_t \in (0,1)\) = forget gate; \(\mathbf{h}_{t-1}\) = previous hidden state; \(\mathbf{x}_t\) = input; \(\sigma\) = sigmoid.
Intuition: Controls how much of the cell state to retain (1=keep all, 0=forget all). A gate close to 0 resets memory; close to 1 preserves it. Key for learning long-range dependencies.

Q: What are the LSTM gate equations?
A: \[i_t=\sigma(W_i[h,x]+b_i),\; f_t=\sigma(W_f[h,x]+b_f),\; o_t=\sigma(W_o[h,x]+b_o),\; \tilde{C}_t=\tanh(W_C[h,x]+b_C)\]
Symbols: \(i_t\)=input gate; \(f_t\)=forget gate; \(o_t\)=output gate; \(\tilde{C}_t\)=candidate cell state.
Intuition: Four gates control information flow through the cell state. Input gate decides what to add; forget gate what to remove; output gate what to read; cell candidate what to potentially write.

Q: What is the LSTM cell state update?
A: \[C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t\]
Symbols: \(C_t\) = cell state; \(\odot\) = element-wise multiply; forget past, add new candidate.
Intuition: The cell state is a 'conveyor belt' — forget gate partially erases old memory, input gate adds new candidate memory. This additive update enables gradients to flow through time without vanishing.

Q: What is the gradient of binary cross-entropy w.r.t. the logit z?
A: \[\frac{\partial \mathcal{L}}{\partial z} = \sigma(z) - y\]
Symbols: \(z\) = logit (pre-sigmoid); \(\sigma(z)\) = predicted probability; \(y\) = binary label.
Intuition: Same clean form as softmax+CE: prediction minus target. This is why sigmoid+BCE is the standard for binary classification — gradient is always in [−1,1] and zero only at the perfect prediction.

Q: What is the number of parameters in a transformer's attention head?
A: \[|W_Q|+|W_K|+|W_V|+|W_O| = 4d_{\text{model}}^2\]
Symbols: \(W_Q, W_K, W_V \in \mathbb{R}^{d_{\text{model}}\times d_k}\); \(W_O \in \mathbb{R}^{h d_k \times d_{\text{model}}}\); \(h d_k = d_{\text{model}}\).
Intuition: Multi-head attention has four weight matrices each of size d²_model → 4d²_model params. For d=768 (BERT-base), that's ≈2.4M per layer for attention alone.

Q: What is the KL divergence term in a variational autoencoder (VAE)?
A: \[\mathcal{L}_{\mathrm{VAE}} = \mathbb{E}_{q_\phi}[\log p_\theta(x|z)] - D_{\mathrm{KL}}(q_\phi(z|x)\|p(z))\]
Symbols: \(q_\phi(z|x)\) = encoder; \(p_\theta(x|z)\) = decoder; \(p(z)=\mathcal{N}(0,I)\) = prior.
Intuition: Two terms: reconstruction loss (how well decoder reconstructs x) − KL regularization (how close encoder posterior is to prior). The KL term prevents encoder from ignoring the prior and ensures the latent space is smooth.

Q: What is the evidence lower bound (ELBO) in terms of reconstruction and KL?
A: \[\mathcal{L} = \mathbb{E}_{q_\phi}[\log p_\theta(x|z)] - D_{\mathrm{KL}}(q_\phi(z|x)\|p(z)) \leq \log p_\theta(x)\]
Symbols: \(\leq \log p_\theta(x)\) = ELBO bounds the marginal log-likelihood from below.
Intuition: Maximizing ELBO is equivalent to simultaneously maximizing reconstruction likelihood and minimizing KL to prior. The gap between ELBO and log p(x) equals KL(q||p(z|x)) — the quality of the variational approximation.

Q: What is the expressiveness of a network with k layers of width n?
A: \[\text{Regions created by ReLU network} = O\!\left(\left(\frac{N}{k}\right)^{(k-1)d}\right)\]
Symbols: \(N\) = total neurons; \(d\) = input dimension; linear regions partition input space.
Intuition: ReLU networks compute piecewise-linear functions. More depth creates exponentially more linear regions than more width (for fixed parameter count). This explains why deep networks work better than wide shallow ones.

Q: What is the effective receptive field and how does it grow with depth?
A: \[\text{ERF} \approx O(\sqrt{L \cdot k^2})\]
Symbols: \(L\) = depth; \(k\) = kernel size; effective RF grows as √L (Gaussian approximation).
Intuition: Though the theoretical RF grows linearly with depth, the effective RF (where gradients are significant) grows sub-linearly. Central pixels receive much more gradient than peripheral ones. Dilated convolutions efficiently increase RF.

Q: What is the gradient with respect to the batch normalization scale parameter γ?
A: \[\frac{\partial \mathcal{L}}{\partial \gamma} = \sum_{i \in \mathcal{B}} \frac{\partial \mathcal{L}}{\partial y_i} \hat{x}_i\]
Symbols: \(\hat{x}_i\) = normalized activation; \(y_i = \gamma\hat{x}_i + \beta\).
Intuition: The gradient of γ is the sum of upstream gradients weighted by the normalized activations. Analogously, ∂L/∂β = Σ ∂L/∂y_i. These learned parameters allow BN to recover the original layer's representational power.

Q: What is pre-activation vs post-activation residual design?
A: \[\text{Post-act: } y = \sigma(F(x)+x); \quad \text{Pre-act: } y = F(\text{BN}(\sigma(x))) + x\]
Symbols: Pre-activation ResNet moves BN and σ before the weight layers.
Intuition: He et al. 2016 found pre-activation (BN+ReLU before weights) allows better gradient flow — the skip connection is a clean identity path. Standard post-activation forces the activation through the non-linearity on the main path.

Q: What is the implicit bias of gradient descent on linear networks?
A: \[\hat{W} = \arg\min_W \|W\|_* \text{ s.t. } WX = Y\]
Symbols: \(\|\cdot\|_*\) = nuclear norm; minimum nuclear norm solution among all zero-training-error solutions.
Intuition: GD on overparameterized linear networks (with zero loss) converges to the minimum nuclear norm solution — implicitly low-rank. This implicit regularization explains generalization without explicit penalties.

Q: What is the NTK regime vs feature learning regime?
A: \[\text{NTK: features fixed, only last layer trained; Feature learning: whole network adapts}\]
Symbols: NTK regime occurs at infinite width or very large learning rate; feature learning at finite width.
Intuition: In the NTK (lazy training) regime, the network barely moves from initialization — acts like a kernel machine. In the feature learning (catapult/rich) regime, hidden representations adapt, enabling better generalization on structured data.

Q: What is gradient signal-to-noise ratio in deep networks?
A: \[\mathrm{SNR}_l = \frac{\|\mathbb{E}[\mathbf{g}_l]\|^2}{\mathrm{Var}[\mathbf{g}_l]}\]
Symbols: \(\mathbf{g}_l\) = gradient at layer l; low SNR means noisy gradients — hard to train.
Intuition: Early layers often have low gradient SNR (small signal, large noise) — making their updates unreliable. Batch normalization and residual connections improve SNR in deep networks by maintaining gradient magnitude.

Q: What is the attention bias in BERT (learned positional embedding)?
A: \[\mathbf{e}_{\text{pos},i} \in \mathbb{R}^{d_{\text{model}}} \text{ learned for each position } i\]
Symbols: \(d_{\text{model}}\) = embedding dimension; maximum positions typically 512 or 2048.
Intuition: Unlike sinusoidal PE (fixed), BERT uses learned position embeddings. Each position gets a trainable vector added to the token embedding. Flexible but limited to the maximum training length.

Q: What is the output of a linear classifier layer?
A: \[\hat{\mathbf{y}} = W_{\text{cls}}\mathbf{h} + \mathbf{b}_{\text{cls}}, \quad W_{\text{cls}} \in \mathbb{R}^{K\times d}\]
Symbols: \(\mathbf{h}\) = final hidden representation; \(K\) = number of classes; \(d\) = model dimension.
Intuition: The classification head projects the final representation to K logits. Then softmax converts to probabilities. This is the standard final layer for classification on top of any backbone.

Q: What is the Query projection matrix W_Q in a transformer?
A: \[Q = X W_Q\]
Symbols: \(X \in \mathbb{R}^{n \times d_{\text{model}}}\) = input token embedding matrix; \(W_Q \in \mathbb{R}^{d_{\text{model}} \times d_k}\) = learned query projection; \(Q \in \mathbb{R}^{n \times d_k}\) = resulting query matrix.
Intuition: Each token creates a "query" vector by projecting its embedding through \(W_Q\), representing what information that token is looking for among all other tokens.

Q: What is the Key projection matrix W_K in a transformer?
A: \[K = X W_K\]
Symbols: \(X \in \mathbb{R}^{n \times d_{\text{model}}}\) = input matrix; \(W_K \in \mathbb{R}^{d_{\text{model}} \times d_k}\) = learned key projection; \(K \in \mathbb{R}^{n \times d_k}\) = key matrix.
Intuition: Each token creates a "key" vector via \(W_K\) advertising what information it contains; queries are compared against all keys to compute compatibility scores.

Q: What is the Value projection matrix W_V in a transformer?
A: \[V = X W_V\]
Symbols: \(X \in \mathbb{R}^{n \times d_{\text{model}}}\) = input matrix; \(W_V \in \mathbb{R}^{d_{\text{model}} \times d_v}\) = learned value projection; \(V \in \mathbb{R}^{n \times d_v}\) = value matrix.
Intuition: Each token creates a "value" vector via \(W_V\) representing the content to be aggregated; after attention weights are computed over keys, the values are blended according to those weights.

Q: What are the three learned projection matrices in a single attention head?
A: \[Q = XW_Q,\quad K = XW_K,\quad V = XW_V\]
Symbols: \(W_Q, W_K \in \mathbb{R}^{d_{\text{model}} \times d_k}\); \(W_V \in \mathbb{R}^{d_{\text{model}} \times d_v}\); all three are learned parameters updated during training.
Intuition: The three projections decompose each token's representation into a query (what to look for), a key (what it contains), and a value (what to output), enabling flexible content-based retrieval.

Q: What is the formula for scaled dot-product attention?
A: \[\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V\]
Symbols: \(Q\) = query matrix; \(K\) = key matrix; \(V\) = value matrix; \(d_k\) = key/query dimension; \(\sqrt{d_k}\) = scaling factor to control magnitude.
Intuition: Queries are matched against all keys via dot products, scaled to prevent softmax saturation, then the resulting probabilities weight-average the values to produce each token's output.

Q: What is the output shape of scaled dot-product attention given n queries and m key-value pairs?
A: \[\text{Attention}(Q,K,V) \in \mathbb{R}^{n \times d_v}\]
Symbols: \(n\) = number of queries; \(m\) = number of key-value pairs; \(Q \in \mathbb{R}^{n \times d_k}\), \(K \in \mathbb{R}^{m \times d_k}\), \(V \in \mathbb{R}^{m \times d_v}\).
Intuition: Each of the \(n\) queries produces one \(d_v\)-dimensional output vector, computed as an attention-weighted sum over the \(m\) value vectors.

Q: How is the attention weight matrix A computed in scaled dot-product attention?
A: \[A = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) \in \mathbb{R}^{n \times m}\]
Symbols: \(Q \in \mathbb{R}^{n \times d_k}\) = queries; \(K \in \mathbb{R}^{m \times d_k}\) = keys; \(A_{ij}\) = how much query \(i\) attends to key \(j\) (row sums to 1).
Intuition: Row \(i\) of \(A\) is a probability distribution over all \(m\) keys, indicating which value vectors query \(i\) should aggregate and in what proportion.

Q: Why do we scale dot products by \(1/\sqrt{d_k}\) in attention?
A: \[\text{Var}(q \cdot k) = d_k \;\Longrightarrow\; \text{Std}(q \cdot k) = \sqrt{d_k}\]
Symbols: \(q, k \in \mathbb{R}^{d_k}\) = query and key vectors with unit-variance components; variance of their dot product grows linearly with \(d_k\); dividing by \(\sqrt{d_k}\) restores unit variance.
Intuition: Without scaling, large \(d_k\) produces large dot products that push softmax into saturation (near one-hot), killing gradients; dividing by \(\sqrt{d_k}\) keeps the inputs to softmax in a stable range.

Q: What happens to the softmax distribution when attention scores are very large?
A: \[\text{softmax}(z_i) = \frac{e^{z_i}}{\sum_j e^{z_j}} \;\xrightarrow{\,z \to \infty\,}\; \text{one-hot}\]
Symbols: \(z_i\) = raw attention score; as scores grow large in magnitude, one entry dominates the exponential, making the distribution extremely peaked with near-zero gradients elsewhere.
Intuition: Very large scores make softmax almost one-hot, effectively hard-assigning attention and zeroing out gradients for all non-maximum positions; scaling by \(1/\sqrt{d_k}\) prevents this regime during training.

Q: How is the raw attention score matrix computed before softmax?
A: \[S = \frac{QK^\top}{\sqrt{d_k}} \in \mathbb{R}^{n \times m}\]
Symbols: \(Q \in \mathbb{R}^{n \times d_k}\) = query matrix; \(K \in \mathbb{R}^{m \times d_k}\) = key matrix; \(S_{ij} = \frac{q_i \cdot k_j}{\sqrt{d_k}}\) = scaled similarity score between query \(i\) and key \(j\).
Intuition: \(S_{ij}\) measures how compatible query \(i\) is with key \(j\); higher scores mean query \(i\) will incorporate more of value \(j\) in its output.

Q: What does entry \(S_{ij}\) of the attention score matrix represent?
A: \[S_{ij} = \frac{q_i^\top k_j}{\sqrt{d_k}}\]
Symbols: \(q_i \in \mathbb{R}^{d_k}\) = query vector for token \(i\); \(k_j \in \mathbb{R}^{d_k}\) = key vector for token \(j\); the scalar result measures alignment in the projected subspace.
Intuition: This score quantifies how relevant token \(j\) is to the information request of token \(i\); after row-wise softmax it becomes the fraction of \(v_j\) mixed into token \(i\)'s output.

Q: How is softmax applied to produce attention weights from the score matrix?
A: \[A_{i,\cdot} = \text{softmax}(S_{i,\cdot}),\quad \text{softmax}(z)_j = \frac{e^{z_j}}{\sum_{k} e^{z_k}}\]
Symbols: \(S_{i,\cdot}\) = row \(i\) of the score matrix (scores from query \(i\) to all keys); \(A_{i,\cdot}\) = resulting probability distribution (non-negative, sums to 1).
Intuition: Softmax converts raw similarities into a valid probability distribution per query so the output is a proper convex combination of value vectors.

Q: What numerical trick stabilizes softmax computation in attention?
A: \[\text{softmax}(z)_j = \frac{e^{z_j - \max_k z_k}}{\sum_{k} e^{z_k - \max_k z_k}}\]
Symbols: \(z_j\) = attention score; \(\max_k z_k\) = row maximum subtracted before exponentiation; the result is mathematically identical to standard softmax but numerically stable.
Intuition: Subtracting the row maximum ensures the largest exponent is \(e^0 = 1\), preventing overflow from very large scores; all deep learning frameworks apply this trick implicitly.

Q: What is self-attention and how are Q, K, V formed?
A: \[Q = XW_Q,\quad K = XW_K,\quad V = XW_V\]
Symbols: \(X \in \mathbb{R}^{n \times d_{\text{model}}}\) = the same input sequence used to produce all three matrices; \(W_Q, W_K, W_V\) = learned projections.
Intuition: In self-attention all three matrices Q, K, V derive from the same sequence \(X\), so every token can attend to every other token in the same sequence, building contextual representations.

Q: What is the output of a self-attention layer for a sequence of n tokens?
A: \[Z = \text{Attention}(XW_Q,\, XW_K,\, XW_V) \in \mathbb{R}^{n \times d_v}\]
Symbols: \(X \in \mathbb{R}^{n \times d_{\text{model}}}\) = input; \(Z_i\) = new representation for token \(i\) as an attention-weighted mixture of all value vectors.
Intuition: Each token's output is a context-aware blend of all other tokens' values, allowing the model to capture global dependencies across the entire sequence in a single layer.

Q: What distinguishes self-attention from a standard fully-connected layer?
A: \[Z_i = \sum_j A_{ij}\, v_j,\quad A_{ij} = \text{softmax}\!\left(\frac{q_i \cdot k_j}{\sqrt{d_k}}\right)\]
Symbols: \(A_{ij}\) = input-dependent weight (recomputed per input, not a fixed parameter); \(v_j\) = value vector of token \(j\).
Intuition: A linear layer uses fixed learned weights; self-attention weights are computed on-the-fly from the input content, so the mixing pattern adapts dynamically to each input sequence.

Q: What is cross-attention in a transformer decoder?
A: \[Q = H_{\text{dec}} W_Q,\quad K = H_{\text{enc}} W_K,\quad V = H_{\text{enc}} W_V\]
Symbols: \(H_{\text{dec}}\) = decoder hidden states (source of queries); \(H_{\text{enc}}\) = encoder hidden states (source of keys and values); queries come from one sequence, K/V from another.
Intuition: The decoder queries the encoder's representations — each decoder position asks which encoder positions are most relevant, then aggregates encoder values accordingly to condition generation on the source.

Q: How does cross-attention differ from self-attention?
A: \[\text{Self: } Q,K,V \leftarrow X;\quad \text{Cross: } Q \leftarrow Y,\; K,V \leftarrow X\]
Symbols: \(X\) = source sequence (e.g., encoder output); \(Y\) = target sequence (e.g., partial decoder output); arrows show which sequence provides each projection.
Intuition: Self-attention gathers context within one sequence; cross-attention lets one sequence (the decoder) attend to a different sequence (the encoder), enabling alignment between source and target.

Q: What is multi-head attention?
A: \[\text{MultiHead}(Q,K,V) = \text{Concat}(\text{head}_1,\ldots,\text{head}_h)\,W^O\]
Symbols: \(\text{head}_i = \text{Attention}(QW_Q^i, KW_K^i, VW_V^i)\); \(h\) = number of heads; \(W^O \in \mathbb{R}^{hd_v \times d_{\text{model}}}\) = output projection matrix.
Intuition: Running \(h\) attention heads in parallel lets the model attend to different representational subspaces simultaneously; the output projection then merges all heads back to the model dimension.

Q: How are per-head projection dimensions chosen in multi-head attention?
A: \[d_k = d_v = \frac{d_{\text{model}}}{h}\]
Symbols: \(d_{\text{model}}\) = full model embedding dimension; \(h\) = number of heads; \(d_k\) = key/query dimension per head; \(d_v\) = value dimension per head.
Intuition: Splitting \(d_{\text{model}}\) evenly across \(h\) heads keeps total computation comparable to single-head attention while allowing multiple parallel subspace projections.

Q: What role does the output projection \(W^O\) play in multi-head attention?
A: \[W^O \in \mathbb{R}^{h d_v \times d_{\text{model}}}\]
Symbols: \(h d_v\) = total concatenated dimension from all \(h\) heads; \(d_{\text{model}}\) = target output dimension; \(W^O\) = learned linear map applied after concatenation.
Intuition: After stacking all head outputs, \(W^O\) linearly mixes across heads, letting the model learn how to combine the diverse information captured in different subspaces into one unified representation.

Q: Write out the full multi-head attention formula with per-head projection matrices.
A: \[\text{head}_i = \text{Attention}(QW_Q^i,\, KW_K^i,\, VW_V^i),\quad W_Q^i,W_K^i \in \mathbb{R}^{d_{\text{model}}\times d_k},\; W_V^i \in \mathbb{R}^{d_{\text{model}}\times d_v}\]
Symbols: \(W_Q^i, W_K^i, W_V^i\) = distinct learned projections for head \(i\); each head operates in its own \(d_k\)-dimensional subspace independently.
Intuition: Each head projects Q, K, V into a different subspace and computes attention independently, forcing diversity across heads; the outputs are then concatenated and projected.

Q: How many total parameters do the projection matrices contribute in multi-head attention (with \(d_k = d_v = d_{\text{model}}/h\))?
A: \[h\bigl(d_{\text{model}} d_k + d_{\text{model}} d_k + d_{\text{model}} d_v\bigr) + h d_v \cdot d_{\text{model}} = 4\,d_{\text{model}}^2\]
Symbols: \(h\) = heads; each \(W_Q^i, W_K^i, W_V^i\) contributes \(d_{\text{model}} \times d_k\) or \(d_{\text{model}} \times d_v\) params; \(W^O\) contributes \(hd_v \times d_{\text{model}}\).
Intuition: Multi-head attention with equal head-dimension splitting has exactly \(4d_{\text{model}}^2\) parameters — the same as four square weight matrices — regardless of the number of heads.

Q: What is causal (masked) self-attention and why is it needed in decoders?
A: \[S_{ij}^{\text{masked}} = \begin{cases} S_{ij} & j \le i \\ -\infty & j > i \end{cases}\]
Symbols: \(S_{ij}\) = raw score from position \(i\) to \(j\); setting future positions to \(-\infty\) forces their softmax weight to 0; \(j > i\) = future positions.
Intuition: Autoregressive models must not use future tokens during training or inference; masking future positions with \(-\infty\) before softmax enforces that position \(i\) only attends to positions \(\le i\).

Q: How is the causal mask implemented as a matrix addition?
A: \[S^{\text{masked}} = S + M,\quad M_{ij} = \begin{cases} 0 & j \le i \\ -\infty & j > i \end{cases}\]
Symbols: \(S \in \mathbb{R}^{n \times n}\) = raw score matrix; \(M\) = upper-triangular mask (zeros on/below diagonal, \(-\infty\) above); added element-wise before softmax.
Intuition: The mask \(M\) is a fixed lower-triangular binary structure added to scores; \(-\infty\) entries collapse to 0 after softmax, cleanly zeroing out all future-position attention weights.

Q: What is the structure of the attention weight matrix in causal self-attention on n tokens?
A: \[A = \text{softmax}(S^{\text{masked}}) \in \mathbb{R}^{n \times n},\quad A_{ij} = 0 \text{ for all } j > i\]
Symbols: \(A_{ij}\) = attention weight from token \(i\) to token \(j\); upper triangle is exactly 0; each row of the lower triangle forms a valid probability distribution.
Intuition: The weight matrix is strictly lower-triangular (including the diagonal), so each token aggregates only itself and past tokens — enforcing the autoregressive property required for language modelling.

Q: What is the sinusoidal positional encoding formula for even-indexed dimensions?
A: \[\text{PE}(\text{pos},\, 2i) = \sin\!\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)\]
Symbols: \(\text{pos}\) = token position (0-indexed); \(i\) = dimension pair index; \(d_{\text{model}}\) = embedding size; \(10000^{2i/d_{\text{model}}}\) = wavelength scale factor forming a geometric progression.
Intuition: Each dimension encodes position with a sine wave at a different frequency; small \(i\) gives fast oscillation (local structure), large \(i\) gives slow oscillation (global structure), together uniquely encoding every position.

Q: What is the sinusoidal positional encoding formula for odd-indexed dimensions?
A: \[\text{PE}(\text{pos},\, 2i+1) = \cos\!\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)\]
Symbols: \(\text{pos}\) = sequence position; \(i\) = dimension pair index; cosine is paired with sine at the same frequency to form a 2D rotation representation.
Intuition: Pairing sin and cos at each frequency means the encoding for \(\text{pos}+k\) can be expressed as a linear (rotation) transformation of the encoding for \(\text{pos}\), letting the model generalise to relative offsets.

Q: How is positional encoding combined with token embeddings?
A: \[\tilde{X}_i = E_i + \text{PE}(i) \in \mathbb{R}^{d_{\text{model}}}\]
Symbols: \(E_i\) = token embedding at position \(i\); \(\text{PE}(i) \in \mathbb{R}^{d_{\text{model}}}\) = positional encoding vector (same dimension); summed element-wise before entering the first layer.
Intuition: Transformers have no recurrence or convolution, so position information must be injected explicitly; adding PE to the embedding informs all subsequent attention and FFN layers of each token's location.

Q: What is the feed-forward sublayer formula in a transformer block?
A: \[\text{FFN}(x) = \max(0,\, xW_1 + b_1)W_2 + b_2\]
Symbols: \(x \in \mathbb{R}^{d_{\text{model}}}\) = per-token input; \(W_1 \in \mathbb{R}^{d_{\text{model}} \times d_{ff}}\); \(W_2 \in \mathbb{R}^{d_{ff} \times d_{\text{model}}}\); \(d_{ff}\) = inner dimension; \(\max(0,\cdot)\) = ReLU activation.
Intuition: The two-layer MLP with ReLU adds per-token nonlinear capacity; cross-token mixing is left to attention, while the FFN independently refines each token's representation.

Q: What is the typical ratio of the FFN inner dimension \(d_{ff}\) to the model dimension \(d_{\text{model}}\)?
A: \[d_{ff} = 4 \cdot d_{\text{model}}\]
Symbols: \(d_{\text{model}}\) = model dimension (e.g., 512); \(d_{ff}\) = inner FFN dimension (e.g., 2048); the 4× expansion is the default in the original Transformer paper.
Intuition: The wide hidden layer can store many features or "memories"; the first projection expands to a rich feature space and the second projects back, providing nonlinear capacity without increasing the residual stream dimension.

Q: Why does the FFN sublayer operate position-wise (independently per token)?
A: \[\text{FFN}(X) = \bigl[\text{FFN}(x_1);\, \text{FFN}(x_2);\,\ldots;\, \text{FFN}(x_n)\bigr]\]
Symbols: \(X \in \mathbb{R}^{n \times d_{\text{model}}}\) = sequence; the same weight matrices \(W_1, W_2\) are applied independently to each \(x_i\); no cross-position interaction occurs.
Intuition: Cross-position mixing is already handled by attention; the FFN's role is purely to transform each token's representation nonlinearly, akin to a per-position lookup or feature refinement step.

Q: What is the residual connection formula in a transformer block?
A: \[y = x + \text{Sublayer}(x)\]
Symbols: \(x\) = input to the sublayer; \(\text{Sublayer}(x)\) = output of the attention or FFN layer; \(y\) = output with skip connection bypassing the sublayer.
Intuition: The shortcut path allows gradients to flow directly to earlier layers without passing through the sublayer, alleviating vanishing gradients and letting the model learn residual corrections rather than full transformations.

Q: What is the Pre-Layer Norm (Pre-LN) formula used in modern transformers?
A: \[y = x + \text{Sublayer}\!\bigl(\text{LayerNorm}(x)\bigr)\]
Symbols: \(\text{LayerNorm}\) normalises the input before entering the sublayer; the unnormalized \(x\) is added back via the residual; contrast with Post-LN where norm follows the sublayer.
Intuition: Normalising before the sublayer (Pre-LN, used in GPT-2 and most modern LLMs) improves training stability and reduces sensitivity to learning rate compared to the original Post-LN Transformer.

Q: What does Layer Normalization compute?
A: \[\text{LayerNorm}(x) = \gamma \odot \frac{x - \mu}{\sigma + \epsilon} + \beta\]
Symbols: \(\mu = \frac{1}{d}\sum_i x_i\) = feature mean; \(\sigma = \sqrt{\frac{1}{d}\sum_i (x_i-\mu)^2}\) = feature std; \(\gamma, \beta \in \mathbb{R}^d\) = learned scale and shift; \(\epsilon\) = small constant for numerical stability.
Intuition: LayerNorm standardises each token's \(d\)-dimensional vector across its own features (not across the batch), then rescales with learned affine parameters, stabilising the distribution of activations entering each sublayer.

Q: What is the overall structure of the Transformer encoder-decoder architecture?
A: \[\underbrace{[\text{Self-Attn} \!\to\! \text{FFN}]^{N}}_{\text{Encoder}} \;\xrightarrow{H_{\text{enc}}}\; \underbrace{[\text{Masked Self-Attn} \!\to\! \text{Cross-Attn} \!\to\! \text{FFN}]^{M}}_{\text{Decoder}}\]
Symbols: \(N\) = encoder layers; \(M\) = decoder layers; \(H_{\text{enc}}\) = encoder output fed as K, V to each decoder cross-attention sublayer.
Intuition: The encoder reads the full source with bidirectional self-attention; the decoder autoregressively generates the target using causal self-attention plus cross-attention over the encoder output.

Q: What is the computational complexity of self-attention with respect to sequence length n and dimension d?
A: \[\mathcal{O}(n^2 d)\]
Symbols: \(n\) = sequence length; \(d\) = model/key dimension; \(n^2\) arises from computing all \(n \times n\) pairwise attention scores; \(d\) from the cost of each dot product.
Intuition: Every token must attend to every other token (\(n^2\) pairs), each requiring a \(d_k\)-dimensional dot product; this quadratic scaling in sequence length is the main bottleneck for long-context models.

Q: What is a key-value (KV) cache and how does it speed up autoregressive decoding?
A: \[K_{1:t} = [k_1, \ldots, k_t],\quad V_{1:t} = [v_1, \ldots, v_t]\quad\text{(cached across steps)}\]
Symbols: \(k_i, v_i\) = key and value projections for token \(i\); at generation step \(t\), only the new token's projections are computed and appended; past projections are reused from cache.
Intuition: Past tokens' K and V never change during generation; caching them avoids recomputing the full sequence at every step, reducing per-step complexity from \(\mathcal{O}(t^2 d)\) to \(\mathcal{O}(t d)\) and making inference practical.

Q: What are relative positional encodings and how do they differ from absolute sinusoidal encodings?
A: \[S_{ij}^{\text{rel}} = \frac{(x_i W_Q)(x_j W_K + r_{i-j} W_K^R)^\top}{\sqrt{d_k}}\]
Symbols: \(r_{i-j}\) = relative position embedding depending only on offset \(i-j\); \(W_K^R\) = learned projection for relative bias; absolute PE adds a fixed vector to embeddings regardless of context.
Intuition: Relative PE conditions attention scores on the distance between tokens rather than absolute positions, giving better length generalisation and the inductive bias that nearby tokens are more related regardless of where they appear.

Q: What is Flash Attention and why is it faster than standard attention?
A: \[\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V \quad\text{computed without materialising }S \in \mathbb{R}^{n \times n}\]
Symbols: \(S = QK^\top/\sqrt{d_k}\) = full score matrix (\(n^2\) entries); Flash Attention tiles Q, K, V into SRAM blocks and fuses score computation, softmax, and value aggregation into one GPU kernel without writing \(S\) to HBM.
Intuition: Standard attention reads/writes the \(n \times n\) matrix to slow GPU HBM; Flash Attention keeps live data in fast on-chip SRAM by processing tiles, reducing HBM traffic from \(\mathcal{O}(n^2)\) to \(\mathcal{O}(n)\) and giving large wall-clock speedups on modern hardware.