AI Math (500 Questions) Practice Exam

Question 1 AI Math (500 Questions)

What is the Pre-Layer Norm (Pre-LN) formula used in modern transformers?

\[y = x + \text{Sublayer}\!\bigl(\text{LayerNorm}(x)\bigr)\]
Symbols: \(\text{LayerNorm}\) normalises the input before entering the sublayer; the unnormalized \(x\) is added back via the residual; contrast with Post-LN where norm follows the sublayer.
Intuition: Normalising before the sublayer (Pre-LN, used in GPT-2 and most modern LLMs) improves training stability and reduces sensitivity to learning rate compared to the original Post-LN Transformer. \[\mathrm{proj}_{\mathbf{u}}\mathbf{v} = \frac{\mathbf{v} \cdot \mathbf{u}}{\|\mathbf{u}\|^2}\mathbf{u}\]
Symbols: \(\mathbf{u}\) = direction to project onto; \(\mathbf{v} \cdot \mathbf{u}\) = dot product.
Intuition: The 'shadow' of v onto u. The scalar coefficient gives how much of v lies in the direction of u. \[\theta_{t+1} = \theta_t - [H(\theta_t)]^{-1}\nabla\mathcal{L}(\theta_t)\]
Symbols: \(H = \nabla^2\mathcal{L}\) = Hessian; \(H^{-1}\) = inverse Hessian.
Intuition: Uses curvature (Hessian) to scale the gradient — takes a large step in flat directions and small step in steep directions. Quadratic convergence near the optimum. Too expensive for DL (O(n²) memory, O(n³) inversion). \[\nabla f(\mathbf{x}) \text{ points in the direction of steepest ascent of } f\]
Symbols: \(-\nabla f(\mathbf{x})\) = direction of steepest descent (used in gradient descent).
Intuition: If you stand on a hilly surface, the gradient points uphill. Moving in the negative gradient direction descends fastest — the foundation of gradient-based optimization.

Question 2 AI Math (500 Questions)

What is the universal approximation theorem?

\[\hat{\theta}_{\mathrm{MAP}} = \arg\min_\theta \left[-\sum_i \log p(x_i|\theta) + \lambda\|\theta\|^2\right]\]
Symbols: \(\lambda\|\theta\|^2\) = L2 regularization term; \(\lambda\) = regularization strength.
Intuition: A Gaussian prior \(p(\theta) \propto e^{-\lambda\|\theta\|^2}\) corresponds to L2 regularization. MAP = MLE + regularizer — Bayesian interpretation of weight decay. \[\min_{p(z|x)} I(X;Z) - \beta I(Z;Y)\]
Symbols: \(Z\) = compressed representation; \(Y\) = target; \(\beta\) = tradeoff; minimize compression while preserving prediction.
Intuition: Find the minimal sufficient statistic for predicting Y from X. The tradeoff parameter β controls compression vs. accuracy. Theoretical lens on what representations DNNs learn. \[\forall\, f \in C(\mathbb{R}^n,\mathbb{R}),\, \epsilon>0:\; \exists \text{ NN with 1 hidden layer s.t. } \sup|\hat{f}-f|<\epsilon\]
Symbols: \(C(\mathbb{R}^n,\mathbb{R})\) = continuous functions; width may need to be exponentially large.
Intuition: A single hidden layer with enough neurons can approximate any continuous function to arbitrary precision. Doesn't say how many neurons or how to find them — provides existence, not efficiency. \[I(x) = \begin{cases}-\log_2 p(x) & \text{bits (shannons)} \\ -\ln p(x) & \text{nats} \\ -\log_{10} p(x) & \text{hartleys}\end{cases}\]
Symbols: Different log bases give different units but same mathematical structure.
Intuition: In ML, nats (natural log) are most common in theory; bits (log base 2) in communication theory. Conversion: 1 nat = log₂(e) ≈ 1.44 bits.

Question 3 AI Math (500 Questions)

How is the causal mask implemented as a matrix addition?

\[A = A^\top \implies \lambda_i \in \mathbb{R} \text{ and eigenvectors are orthogonal}\]
Symbols: \(A^\top\) = transpose; \(\lambda_i\) = eigenvalues.
Intuition: The spectral theorem: symmetric matrices are always diagonalizable with real eigenvalues and mutually orthogonal eigenvectors. Covariance matrices and Hessians benefit from this. \[\text{Self: } Q,K,V \leftarrow X;\quad \text{Cross: } Q \leftarrow Y,\; K,V \leftarrow X\]
Symbols: \(X\) = source sequence (e.g., encoder output); \(Y\) = target sequence (e.g., partial decoder output); arrows show which sequence provides each projection.
Intuition: Self-attention gathers context within one sequence; cross-attention lets one sequence (the decoder) attend to a different sequence (the encoder), enabling alignment between source and target. \[S^{\text{masked}} = S + M,\quad M_{ij} = \begin{cases} 0 & j \le i \\ -\infty & j > i \end{cases}\]
Symbols: \(S \in \mathbb{R}^{n \times n}\) = raw score matrix; \(M\) = upper-triangular mask (zeros on/below diagonal, \(-\infty\) above); added element-wise before softmax.
Intuition: The mask \(M\) is a fixed lower-triangular binary structure added to scores; \(-\infty\) entries collapse to 0 after softmax, cleanly zeroing out all future-position attention weights. \[\mathcal{L}(\theta_T) - \mathcal{L}(\theta^*) \leq \frac{L\|\theta_0 - \theta^*\|^2}{2T}\]
Symbols: \(T\) = iterations; \(\theta^*\) = optimal parameters; \(L\) = smoothness.
Intuition: The loss excess decreases as O(1/T) — sublinear convergence. For strongly convex functions it's O(ρ^T) (linear/geometric convergence), which is much faster.

Question 4 AI Math (500 Questions)

What is an objective (loss) function in ML optimization?

\[d_{ff} = 4 \cdot d_{\text{model}}\]
Symbols: \(d_{\text{model}}\) = model dimension (e.g., 512); \(d_{ff}\) = inner FFN dimension (e.g., 2048); the 4× expansion is the default in the original Transformer paper.
Intuition: The wide hidden layer can store many features or "memories"; the first projection expands to a rich feature space and the second projects back, providing nonlinear capacity without increasing the residual stream dimension. \[S = \frac{QK^\top}{\sqrt{d_k}} \in \mathbb{R}^{n \times m}\]
Symbols: \(Q \in \mathbb{R}^{n \times d_k}\) = query matrix; \(K \in \mathbb{R}^{m \times d_k}\) = key matrix; \(S_{ij} = \frac{q_i \cdot k_j}{\sqrt{d_k}}\) = scaled similarity score between query \(i\) and key \(j\).
Intuition: \(S_{ij}\) measures how compatible query \(i\) is with key \(j\); higher scores mean query \(i\) will incorporate more of value \(j\) in its output. \[\min_{\theta} \mathcal{L}(\theta) = \frac{1}{N}\sum_{i=1}^N \ell(f_\theta(x_i), y_i)\]
Symbols: \(\theta\) = parameters; \(\ell\) = per-sample loss; \(f_\theta\) = model.
Intuition: The scalar quantity we minimize to train the model. Average per-sample loss over the training set. Different tasks use different loss functions (MSE, CE, hinge). \[D_{\mathrm{KL}}(q\|p) = -\frac{1}{2}\sum_{j=1}^d \left(1 + \log\sigma_j^2 - \mu_j^2 - \sigma_j^2\right)\]
Symbols: \(d\) = latent dimension; \(\mu_j, \sigma_j^2\) = variational parameters per dimension.
Intuition: Closed-form regularization term in VAE training. Pushes posterior toward the standard Gaussian prior — prevents the encoder from ignoring the prior.

Question 5 AI Math (500 Questions)

What is the eigenspace corresponding to an eigenvalue λ?

\[E_\lambda = \ker(A - \lambda I) = \{\mathbf{v} : A\mathbf{v} = \lambda \mathbf{v}\}\]
Symbols: \(E_\lambda\) = eigenspace; \(\ker\) = null space.
Intuition: All eigenvectors for λ (plus the zero vector) form a subspace. Dimension of eigenspace = geometric multiplicity of λ. \[f\!\left(\mathbb{E}[X]\right) \leq \mathbb{E}[f(X)]\]
Symbols: \(X\) = random variable; \(\mathbb{E}\) = expectation.
Intuition: For convex f, the function of the mean ≤ the mean of the function. Used in proving KL divergence ≥ 0, EM algorithm bounds, and many information-theoretic inequalities. \[D_{\mathrm{KL}}(q\|p) = -\frac{1}{2}\sum_{j=1}^d \left(1 + \log\sigma_j^2 - \mu_j^2 - \sigma_j^2\right)\]
Symbols: \(d\) = latent dimension; \(\mu_j, \sigma_j^2\) = variational parameters per dimension.
Intuition: Closed-form regularization term in VAE training. Pushes posterior toward the standard Gaussian prior — prevents the encoder from ignoring the prior. \[\text{Receptive field at layer } l = k + (k-1)(l-1) \text{ (for kernel size k)}\]
Symbols: \(k\) = kernel size; \(l\) = depth; grows linearly with depth for standard convolution.
Intuition: How large a region of the input influences one output neuron. Deeper layers see a larger context. Dilated convolutions grow the receptive field exponentially (used in WaveNet, DeepLab).

Question 6 AI Math (500 Questions)

What are pooling operations in CNNs?

\[f(\lambda\mathbf{x} + (1-\lambda)\mathbf{y}) \leq \lambda f(\mathbf{x}) + (1-\lambda)f(\mathbf{y}), \quad \lambda \in [0,1]\]
Symbols: \(\lambda\) = interpolation parameter; \(\mathbf{x}, \mathbf{y}\) = any two points.
Intuition: The chord between any two points lies above the function. Convex functions have no local minima that aren't global — gradient descent is guaranteed to converge. \[f \text{ convex} \iff \nabla^2 f(\mathbf{x}) \succeq 0 \text{ for all } \mathbf{x}\]
Symbols: \(\nabla^2 f\) = Hessian; \(\succeq 0\) = positive semi-definite.
Intuition: A function is convex iff its Hessian is PSD everywhere — non-negative curvature in all directions. Strictly convex has PD Hessian and a unique global minimum. \[\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V \quad\text{computed without materialising }S \in \mathbb{R}^{n \times n}\]
Symbols: \(S = QK^\top/\sqrt{d_k}\) = full score matrix (\(n^2\) entries); Flash Attention tiles Q, K, V into SRAM blocks and fuses score computation, softmax, and value aggregation into one GPU kernel without writing \(S\) to HBM.
Intuition: Standard attention reads/writes the \(n \times n\) matrix to slow GPU HBM; Flash Attention keeps live data in fast on-chip SRAM by processing tiles, reducing HBM traffic from \(\mathcal{O}(n^2)\) to \(\mathcal{O}(n)\) and giving large wall-clock speedups on modern hardware. \[\text{MaxPool}(X)_{ij} = \max_{(m,n) \in \text{window}} X_{im,jn}; \quad \text{AvgPool} = \text{mean}\]
Symbols: Reduces spatial dimensions; window = pooling kernel size.
Intuition: Downsamples feature maps to reduce spatial resolution and parameter count. Max pooling selects the most active feature in each region (introduces some translation invariance). Average pooling computes the mean.

Question 7 AI Math (500 Questions)

What is the Geometric distribution?

\[P(X=k) = (1-p)^{k-1}p, \quad k=1,2,\ldots, \quad \mathbb{E}[X]=\frac{1}{p}\]
Symbols: \(p\) = success probability; \(k\) = trial on which first success occurs.
Intuition: Number of trials until first success. Memoryless property: P(X>m+n|X>m) = P(X>n). Only discrete memoryless distribution. \[\hat{\theta}_{\mathrm{MAP}} = \arg\min_\theta \left[-\sum_i \log p(x_i|\theta) + \lambda\|\theta\|^2\right]\]
Symbols: \(\lambda\|\theta\|^2\) = L2 regularization term; \(\lambda\) = regularization strength.
Intuition: A Gaussian prior \(p(\theta) \propto e^{-\lambda\|\theta\|^2}\) corresponds to L2 regularization. MAP = MLE + regularizer — Bayesian interpretation of weight decay. \[W_1 = W_2 = \cdots = W_T \text{ (same weights across positions/time)}\]
Symbols: Tied weights; greatly reduces parameter count.
Intuition: CNNs share filter weights across spatial positions. RNNs share weights across time steps. Transformers share attention weights across positions. Reduces parameters, enforces translation/temporal equivariance, improves generalization. \[\frac{d}{dx}[f(g(x))] = f'(g(x)) \cdot g'(x)\]
Symbols: \(f \circ g\) = composite function; \(f'(g(x))\) = outer derivative evaluated at inner function.
Intuition: 'Derivative of outer times derivative of inner.' The most important rule in deep learning — backpropagation is essentially the chain rule applied repeatedly.

Question 8 AI Math (500 Questions)

How is explained variance computed in PCA?

\[\nabla f(\mathbf{x}^*) = \lambda \nabla g(\mathbf{x}^*), \quad g(\mathbf{x}^*) = 0\]
Symbols: \(\lambda\) = Lagrange multiplier; \(g(\mathbf{x})=0\) = equality constraint.
Intuition: At a constrained optimum, the gradients of objective and constraint are parallel — f can't improve without violating the constraint. λ measures the 'exchange rate' between objective and constraint. \[A = U\Sigma V^\top\]
Symbols: \(A \in \mathbb{R}^{m \times n}\); \(U \in \mathbb{R}^{m \times m}\) = left singular vectors (orthogonal); \(\Sigma \in \mathbb{R}^{m \times n}\) = diagonal (singular values \(\sigma_i \geq 0\)); \(V \in \mathbb{R}^{n \times n}\) = right singular vectors (orthogonal).
Intuition: Every matrix decomposes as: rotate (V^T), scale axes (Σ), rotate again (U). Generalizes eigendecomposition to non-square matrices. \[\text{Explained variance ratio}_k = \frac{\sum_{i=1}^k \lambda_i}{\sum_{i=1}^d \lambda_i}\]
Symbols: \(\lambda_i\) = eigenvalues of the covariance matrix sorted descending; \(d\) = total features.
Intuition: The fraction of total variance captured by the top k components. Used to choose k: e.g., keep components explaining 95% of variance. \[\hat{\mathbf{x}} = (A^\top A)^{-1}A^\top \mathbf{b}\]
Symbols: \(A^\top A\) = Gram matrix; \(A^\top \mathbf{b}\) = normal equations right-hand side.
Intuition: Solves Ax ≈ b when no exact solution exists, by projecting b onto col(A). The normal equations \(A^\top A\hat{x} = A^\top b\) characterize the solution.

Question 9 AI Math (500 Questions)

What does entry \(S_{ij}\) of the attention score matrix represent?

\[S_{ij} = \frac{q_i^\top k_j}{\sqrt{d_k}}\]
Symbols: \(q_i \in \mathbb{R}^{d_k}\) = query vector for token \(i\); \(k_j \in \mathbb{R}^{d_k}\) = key vector for token \(j\); the scalar result measures alignment in the projected subspace.
Intuition: This score quantifies how relevant token \(j\) is to the information request of token \(i\); after row-wise softmax it becomes the fraction of \(v_j\) mixed into token \(i\)'s output. \[\hat{\Sigma} = \frac{1}{N-1}\sum_{i=1}^N (\mathbf{x}_i - \bar{\mathbf{x}})(\mathbf{x}_i - \bar{\mathbf{x}})^\top\]
Symbols: \(\bar{\mathbf{x}}\) = sample mean; \(N\) = number of samples; dividing by N-1 gives an unbiased estimator.
Intuition: Empirical estimate of the population covariance matrix. Positive semi-definite; singular if N-1 < d. \[\mathrm{Var}(a^{(l)}) = n_{\text{in}} \cdot \mathrm{Var}(W) \cdot \mathrm{Var}(a^{(l-1)})\]
Symbols: Layer output variance scales with layer width and weight variance.
Intuition: If weight variance is too large, activations explode. Too small, they vanish to zero. Xavier/He set variance to exactly cancel the fan-in scaling, maintaining unit variance throughout the network. \[\mathcal{L}_\delta(r) = \begin{cases}\frac{1}{2}r^2 & |r| \leq \delta \\ \delta|r| - \frac{1}{2}\delta^2 & |r| > \delta\end{cases}\]
Symbols: \(r = y - \hat{y}\) = residual; \(\delta\) = threshold between quadratic and linear regions.
Intuition: Quadratic for small residuals (smooth, fast convergence), linear for large residuals (outlier-robust). Best of MSE and MAE. Used in regression with outliers (DQN Q-value loss).

Question 10 AI Math (500 Questions)

What is the log-normal distribution?

\[A\mathbf{x} = \sum_{j=1}^n x_j \mathbf{a}_j\]
Symbols: \(\mathbf{a}_j\) = j-th column of A; \(x_j\) = j-th component of x.
Intuition: Multiplying a matrix by a vector produces a linear combination of the matrix's columns, weighted by the vector's components. \[\theta_{t+1} = \theta_t - \eta_t \nabla_\theta \ell(f_\theta(x_i), y_i)\]
Symbols: \((x_i, y_i)\) = single randomly sampled training example; noisy gradient estimate.
Intuition: Instead of computing the full gradient over all N examples, use a single random example. Much cheaper per step (O(1) vs O(N)), converges in expectation to a minimum. \[\log X \sim \mathcal{N}(\mu,\sigma^2) \implies \mathbb{E}[X] = e^{\mu + \sigma^2/2}\]
Symbols: \(X > 0\); a positive-valued variable whose log is Gaussian.
Intuition: Models multiplicative processes (growth rates, prices, word frequencies). Heavy right tail — mean > median. Common in language (Zipf's law approximation). \[W_1 = W_2 = \cdots = W_T \text{ (same weights across positions/time)}\]
Symbols: Tied weights; greatly reduces parameter count.
Intuition: CNNs share filter weights across spatial positions. RNNs share weights across time steps. Transformers share attention weights across positions. Reduces parameters, enforces translation/temporal equivariance, improves generalization.

Question 11 AI Math (500 Questions)

What is the Mean Squared Error (MSE) loss?

\[\mathcal{O}(n^2 d)\]
Symbols: \(n\) = sequence length; \(d\) = model/key dimension; \(n^2\) arises from computing all \(n \times n\) pairwise attention scores; \(d\) from the cost of each dot product.
Intuition: Every token must attend to every other token (\(n^2\) pairs), each requiring a \(d_k\)-dimensional dot product; this quadratic scaling in sequence length is the main bottleneck for long-context models. \[y = x + \text{Sublayer}(x)\]
Symbols: \(x\) = input to the sublayer; \(\text{Sublayer}(x)\) = output of the attention or FFN layer; \(y\) = output with skip connection bypassing the sublayer.
Intuition: The shortcut path allows gradients to flow directly to earlier layers without passing through the sublayer, alleviating vanishing gradients and letting the model learn residual corrections rather than full transformations. \[A^k = Q\Lambda^k Q^{-1}, \quad \Lambda^k = \mathrm{diag}(\lambda_1^k,\ldots,\lambda_n^k)\]
Symbols: \(k\) = integer power.
Intuition: Powers of diagonal matrices are trivial — just raise each diagonal element to the power. So eigendecomposition makes matrix powers efficient. \[\mathcal{L}_{\mathrm{MSE}} = \frac{1}{N}\sum_{i=1}^N (y_i - \hat{y}_i)^2\]
Symbols: \(y_i\) = true target; \(\hat{y}_i\) = prediction; squared penalty for each error.
Intuition: Standard regression loss. Penalizes large errors more heavily (squared). MLE for Gaussian noise model. Gradient: 2(ŷ-y)/N. Sensitive to outliers due to squaring.

Question 12 AI Math (500 Questions)

Are eigenvectors corresponding to distinct eigenvalues linearly independent?

\[\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V \quad\text{computed without materialising }S \in \mathbb{R}^{n \times n}\]
Symbols: \(S = QK^\top/\sqrt{d_k}\) = full score matrix (\(n^2\) entries); Flash Attention tiles Q, K, V into SRAM blocks and fuses score computation, softmax, and value aggregation into one GPU kernel without writing \(S\) to HBM.
Intuition: Standard attention reads/writes the \(n \times n\) matrix to slow GPU HBM; Flash Attention keeps live data in fast on-chip SRAM by processing tiles, reducing HBM traffic from \(\mathcal{O}(n^2)\) to \(\mathcal{O}(n)\) and giving large wall-clock speedups on modern hardware. \[\text{if } \|\mathbf{g}\| > \tau: \quad \mathbf{g} \leftarrow \frac{\tau}{\|\mathbf{g}\|}\mathbf{g}\]
Symbols: \(\mathbf{g}\) = gradient vector; \(\tau\) = clipping threshold; \(\|\mathbf{g}\|\) = gradient norm.
Intuition: When gradients explode (huge norms), scale them back to have norm τ while preserving direction. Essential for training RNNs and transformers on long sequences. \[\sum_{t=1}^\infty \eta_t = \infty, \quad \sum_{t=1}^\infty \eta_t^2 < \infty \quad (\text{e.g., } \eta_t = 1/t)\]
Symbols: Robbins-Monro conditions on the learning rate schedule.
Intuition: First condition ensures SGD makes enough total progress to reach the optimum. Second ensures steps decrease fast enough so noise doesn't prevent convergence. Decaying LR is necessary for convergence of SGD. \[\lambda_i \neq \lambda_j \implies \mathbf{v}_i \not\parallel \mathbf{v}_j \text{ and } \{\mathbf{v}_i\} \text{ linearly independent}\]
Symbols: \(\lambda_i\) = distinct eigenvalues; \(\mathbf{v}_i\) = corresponding eigenvectors.
Intuition: Distinct eigenvalues guarantee linearly independent eigenvectors, which is necessary for diagonalization.

Question 13 AI Math (500 Questions)

What is the covariance matrix of a random vector?

\[\eta^* = \arg\max_\eta \left(-\frac{d\mathcal{L}}{d\log\eta}\right)\]
Symbols: Find LR where loss decreases fastest (steepest negative slope on LR vs. loss curve).
Intuition: Run training with exponentially increasing LR for one epoch. Plot LR vs. loss and pick the LR where loss decreases most steeply (just before it starts to diverge). Proposed by Leslie Smith (2015). \[\bar{L}_{\mathrm{Huffman}} < H(X) + 1\]
Symbols: \(\bar{L}\) = average code length; Huffman achieves within 1 bit of entropy.
Intuition: Huffman assigns shorter codes to frequent symbols. Average length is between H(X) and H(X)+1 bits. An optimal prefix-free code. Achieves Shannon's bound up to 1 bit. \[\Sigma = \mathbb{E}[(\mathbf{x}-\boldsymbol{\mu})(\mathbf{x}-\boldsymbol{\mu})^\top]\]
Symbols: \(\boldsymbol{\mu} = \mathbb{E}[\mathbf{x}]\) = mean vector; \(\Sigma_{ij} = \mathrm{Cov}(x_i, x_j)\).
Intuition: Encodes the variance of each feature (diagonal) and co-movement between features (off-diagonal). Always symmetric and positive semi-definite. \[\|\nabla\mathcal{L}(\theta_1) - \nabla\mathcal{L}(\theta_2)\| \leq L\|\theta_1-\theta_2\|\]
Symbols: \(L\) = Lipschitz constant of the gradient; \(H \preceq LI\) (Hessian bounded).
Intuition: The gradient doesn't change too rapidly. For L-smooth functions, GD with learning rate η ≤ 1/L is guaranteed to decrease the loss each step. Relates to the maximum safe learning rate.

Question 14 AI Math (500 Questions)

What is the best rank-k approximation of a matrix?

\[\mathbf{z} = \mu + \sigma \odot \boldsymbol{\epsilon}, \quad \boldsymbol{\epsilon} \sim \mathcal{N}(0, I)\]
Symbols: \(\mu, \sigma\) = learned mean/std; \(\odot\) = element-wise multiply; \(\boldsymbol{\epsilon}\) = noise sample.
Intuition: Makes the sampling operation differentiable by moving the stochasticity to ε (which doesn't depend on parameters). Enables backprop through stochastic nodes in VAEs. \[\text{Full GD: zero variance, but O(N) cost; SGD: high variance, O(1) cost}\]
Symbols: Variance of mini-batch gradient ∝ \(\sigma^2_{\mathrm{grad}}/B\).
Intuition: Larger batches give lower variance gradient estimates but cost more per step. Smaller batches allow more steps per second but noisier updates. Variance decreases as 1/B. \[Q = H_{\text{dec}} W_Q,\quad K = H_{\text{enc}} W_K,\quad V = H_{\text{enc}} W_V\]
Symbols: \(H_{\text{dec}}\) = decoder hidden states (source of queries); \(H_{\text{enc}}\) = encoder hidden states (source of keys and values); queries come from one sequence, K/V from another.
Intuition: The decoder queries the encoder's representations — each decoder position asks which encoder positions are most relevant, then aggregates encoder values accordingly to condition generation on the source. \[A_k = \sum_{i=1}^k \sigma_i \mathbf{u}_i \mathbf{v}_i^\top\]
Symbols: \(\sigma_i\) = i-th singular value; \(\mathbf{u}_i, \mathbf{v}_i\) = i-th left/right singular vectors.
Intuition: The Eckart-Young theorem: the best rank-k approximation (in Frobenius or spectral norm) is obtained by keeping only the top k singular triplets.

Question 15 AI Math (500 Questions)

What is RMSprop?

\[v_t = \rho v_{t-1} + (1-\rho)g_t^2, \quad \theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{v_t}+\epsilon}g_t\]
Symbols: \(v_t\) = exponential moving avg of squared gradients; \(\rho \approx 0.9\) = decay rate.
Intuition: Fixes AdaGrad's diminishing LR by using an exponential moving average instead of sum — only recent gradient magnitudes matter. Non-stationary friendly; widely used in RNNs. \[\text{Too large: } \mathcal{L} \text{ diverges; too small: slow convergence; optimal: } \eta \approx \frac{1}{L}\]
Symbols: \(L\) = Lipschitz constant of gradient; \(\eta_{\mathrm{opt}} = 1/L\) for quadratics.
Intuition: Learning rate is the most important hyperparameter. Too large = gradient descent overshoots and diverges. Too small = too many steps needed. Optimal ≈ inverse of curvature. \[J_{ij} = \frac{\partial f_i}{\partial x_j}, \quad J \in \mathbb{R}^{m \times n}\]
Symbols: \(\mathbf{f}: \mathbb{R}^n \to \mathbb{R}^m\); \(J\) = matrix of all first-order partial derivatives.
Intuition: Generalizes the gradient to vector-valued functions. The Jacobian is the best linear approximation to f near a point — \(\mathbf{f}(\mathbf{x}+\delta) \approx \mathbf{f}(\mathbf{x}) + J\delta\). \[I(x) = \begin{cases}-\log_2 p(x) & \text{bits (shannons)} \\ -\ln p(x) & \text{nats} \\ -\log_{10} p(x) & \text{hartleys}\end{cases}\]
Symbols: Different log bases give different units but same mathematical structure.
Intuition: In ML, nats (natural log) are most common in theory; bits (log base 2) in communication theory. Conversion: 1 nat = log₂(e) ≈ 1.44 bits.

Question 16 AI Math (500 Questions)

What is the computational complexity of self-attention with respect to sequence length n and dimension d?

\[\mathcal{O}(n^2 d)\]
Symbols: \(n\) = sequence length; \(d\) = model/key dimension; \(n^2\) arises from computing all \(n \times n\) pairwise attention scores; \(d\) from the cost of each dot product.
Intuition: Every token must attend to every other token (\(n^2\) pairs), each requiring a \(d_k\)-dimensional dot product; this quadratic scaling in sequence length is the main bottleneck for long-context models. \[P(|X - \mu| \geq k\sigma) \leq \frac{1}{k^2}\]
Symbols: \(\mu = \mathbb{E}[X]\); \(\sigma^2 = \mathrm{Var}(X)\); \(k > 0\).
Intuition: At least 1-1/k² of the probability mass lies within k standard deviations of the mean, for any distribution with finite variance. Weaker than Gaussian but universally applicable. \[H \in \mathbb{R}^{n \times n}, \quad n = \text{number of parameters (millions)}\]
Symbols: \(H^{-1}\) requires \(O(n^2)\) memory and \(O(n^3)\) computation.
Intuition: A network with 1M parameters has a 1M×1M Hessian — impossible to store or invert. Quasi-Newton methods (L-BFGS) approximate H^{-1} with low-rank updates. Used for small-scale problems. \[\mathbf{y} = F(\mathbf{x}, \{W_i\}) + \mathbf{x}\]
Symbols: \(F(\mathbf{x})\) = residual mapping learned by the sub-block; \(\mathbf{x}\) = identity shortcut.
Intuition: The block only needs to learn the residual (difference from input). If the optimal output ≈ input, residual → 0 (easy to learn). Gradient flows directly through the skip connection, solving vanishing gradient in very deep networks.

Question 17 AI Math (500 Questions)

What is the Mahalanobis distance?

\[W \sim \mathcal{N}\!\left(0,\, \frac{2}{n_{\text{in}}}\right)\]
Symbols: \(n_{\text{in}}\) = fan-in; factor of 2 accounts for ReLU killing half the outputs (sets E[a²] = Var(z)).
Intuition: Designed for ReLU activations. ReLU sets half the activations to 0, so needs larger initial variance to maintain signal. Default initialization for ReLU networks in PyTorch. \[\mathbf{a} \perp \mathbf{b} \iff \mathbf{a} \cdot \mathbf{b} = 0\]
Symbols: \(\perp\) = perpendicular symbol.
Intuition: Two vectors are orthogonal (perpendicular) iff their dot product is zero — cos 90° = 0. \[\text{Explained variance ratio}_k = \frac{\sum_{i=1}^k \lambda_i}{\sum_{i=1}^d \lambda_i}\]
Symbols: \(\lambda_i\) = eigenvalues of the covariance matrix sorted descending; \(d\) = total features.
Intuition: The fraction of total variance captured by the top k components. Used to choose k: e.g., keep components explaining 95% of variance. \[d_M(\mathbf{x},\boldsymbol{\mu}) = \sqrt{(\mathbf{x}-\boldsymbol{\mu})^\top \Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu})}\]
Symbols: \(\Sigma^{-1}\) = precision matrix; accounts for correlation and scale.
Intuition: Euclidean distance normalized by the covariance structure. Scale-invariant and accounts for correlated features. Measures how many standard deviations x is from μ in any direction.

Question 18 AI Math (500 Questions)

What are relative positional encodings and how do they differ from absolute sinusoidal encodings?

\[I(x_1 \cap x_2) = I(x_1) + I(x_2) \text{ if } x_1 \perp x_2\]
Symbols: \(\log p(x_1 x_2) = \log p(x_1) + \log p(x_2)\) for independent events.
Intuition: We want information to be additive for independent events. Log of a product = sum of logs, which achieves this. A natural, axiomatically justified choice. \[d_{ff} = 4 \cdot d_{\text{model}}\]
Symbols: \(d_{\text{model}}\) = model dimension (e.g., 512); \(d_{ff}\) = inner FFN dimension (e.g., 2048); the 4× expansion is the default in the original Transformer paper.
Intuition: The wide hidden layer can store many features or "memories"; the first projection expands to a rich feature space and the second projects back, providing nonlinear capacity without increasing the residual stream dimension. \[\mathrm{NMI}(X,Y) = \frac{I(X;Y)}{\sqrt{H(X)H(Y)}} \in [0,1]\]
Symbols: \(I(X;Y)\) = mutual information; normalized to [0,1] by geometric mean of entropies.
Intuition: Normalized version of MI suitable for comparing different pairs of variables. 0 = independent; 1 = perfectly dependent. Used to evaluate clustering quality (compare cluster labels to true labels). \[S_{ij}^{\text{rel}} = \frac{(x_i W_Q)(x_j W_K + r_{i-j} W_K^R)^\top}{\sqrt{d_k}}\]
Symbols: \(r_{i-j}\) = relative position embedding depending only on offset \(i-j\); \(W_K^R\) = learned projection for relative bias; absolute PE adds a fixed vector to embeddings regardless of context.
Intuition: Relative PE conditions attention scores on the distance between tokens rather than absolute positions, giving better length generalisation and the inductive bias that nearby tokens are more related regardless of where they appear.

Question 19 AI Math (500 Questions)

What does Layer Normalization compute?

\[A = LL^\top, \quad L \text{ lower triangular with positive diagonal}\]
Symbols: \(L\) = lower-triangular Cholesky factor.
Intuition: A symmetric PSD matrix can be factored as LL^T. Used to efficiently solve linear systems and sample from multivariate Gaussians. Fails if A is not PSD. \[\hat{\mathbf{y}} = W_{\text{cls}}\mathbf{h} + \mathbf{b}_{\text{cls}}, \quad W_{\text{cls}} \in \mathbb{R}^{K\times d}\]
Symbols: \(\mathbf{h}\) = final hidden representation; \(K\) = number of classes; \(d\) = model dimension.
Intuition: The classification head projects the final representation to K logits. Then softmax converts to probabilities. This is the standard final layer for classification on top of any backbone. \[\sigma_g^2 = \frac{S}{\text{batch size}}, \quad S = \frac{1}{N}\sum_i \|\nabla\ell_i - \nabla\mathcal{L}\|^2\]
Symbols: \(S\) = gradient noise scale; \(N\) = dataset size; \(\nabla\ell_i\) = per-sample gradient.
Intuition: Gradient noise scale S measures the signal-to-noise ratio of mini-batch gradients. OpenAI found that critical batch size ≈ S determines when batch size stops helping — theory for LLM training budget allocation. \[\text{LayerNorm}(x) = \gamma \odot \frac{x - \mu}{\sigma + \epsilon} + \beta\]
Symbols: \(\mu = \frac{1}{d}\sum_i x_i\) = feature mean; \(\sigma = \sqrt{\frac{1}{d}\sum_i (x_i-\mu)^2}\) = feature std; \(\gamma, \beta \in \mathbb{R}^d\) = learned scale and shift; \(\epsilon\) = small constant for numerical stability.
Intuition: LayerNorm standardises each token's \(d\)-dimensional vector across its own features (not across the batch), then rescales with learned affine parameters, stabilising the distribution of activations entering each sublayer.

Question 20 AI Math (500 Questions)

What is the gradient of a composition f(g(x)) where g is a vector function?

\[v_{t+1} = \beta v_t + \nabla\mathcal{L}(\theta_t - \beta\eta v_t), \quad \theta_{t+1} = \theta_t - \eta v_{t+1}\]
Symbols: Gradient evaluated at the 'look-ahead' position \(\theta_t - \beta\eta v_t\).
Intuition: Computes gradient at where we're about to be (after momentum step), not where we are. More 'anticipatory' — slightly faster convergence than standard momentum, especially near minima. \[\frac{\partial f}{\partial \mathbf{x}} = \frac{\partial f}{\partial \mathbf{g}} J_g\]
Symbols: \(J_g = \partial\mathbf{g}/\partial\mathbf{x}\) = Jacobian of g; row-vector times Jacobian.
Intuition: The upstream gradient is left-multiplied by the Jacobian of the inner function. This is the vector-chain-rule used in every DL backward pass. \[\nabla_\phi \mathbb{E}_{q_\phi}[f(z)] = \mathbb{E}_\epsilon[\nabla_\phi f(g(\epsilon,\phi))],\quad z = g(\epsilon,\phi),\;\epsilon \sim p(\epsilon)\]
Symbols: \(\phi\) = variational parameters; \(g\) = differentiable transform; \(\epsilon\) = noise.
Intuition: Moves the gradient inside the expectation by reparameterizing z as a deterministic function of noise ε and parameters φ. Low-variance gradient estimator used in VAE training. \[\nabla_{\mathbf{x}} (\mathbf{x}^\top A \mathbf{x}) = (A + A^\top)\mathbf{x}\]
Symbols: \(A \in \mathbb{R}^{n \times n}\); if A is symmetric: \(\nabla = 2A\mathbf{x}\).
Intuition: For the symmetric case (common in ML), gradient is 2Ax — analogous to the scalar derivative of ax² being 2ax. Key for understanding least squares.

Question 21 AI Math (500 Questions)

What is the binary cross-entropy loss?

\[A = A^\top \iff A_{ij} = A_{ji} \text{ for all } i,j\]
Symbols: \(A^\top\) = transpose of A.
Intuition: A symmetric matrix equals its own transpose. Covariance matrices and Hessians are symmetric. Symmetric matrices always have real eigenvalues. \[\mathcal{L} = -[y\log\hat{y} + (1-y)\log(1-\hat{y})]\]
Symbols: \(y \in \{0,1\}\) = binary label; \(\hat{y} = \sigma(z) \in (0,1)\) = sigmoid output.
Intuition: For binary classification: if y=1, penalizes low ŷ; if y=0, penalizes high ŷ. Negative log-likelihood of Bernoulli distribution. Gradient: ŷ - y. \[\text{softplus}(x) = \ln(1 + e^x) \approx x \text{ for large } x, \approx e^x \text{ for small } x\]
Symbols: Smooth approximation to ReLU.
Intuition: Softplus is the smooth upper envelope of ReLU. Its derivative is the sigmoid: d/dx[softplus(x)] = σ(x). Taylor expansion helps analyze behavior in different regimes. \[X \to Y \to Z \implies I(X;Z) \leq I(X;Y)\]
Symbols: Markov chain X→Y→Z; processing Y to get Z can't add information about X.
Intuition: You can never increase information about X by processing it further. Any transformation discards or preserves, but never creates, information. Key in feature extraction theory.

Question 22 AI Math (500 Questions)

What is the standard deviation?

\[\sigma = \sqrt{\mathrm{Var}(X)}\]
Symbols: \(\sigma\) = standard deviation; same units as X unlike variance.
Intuition: The 'typical' deviation from the mean. More interpretable than variance because it's in the same units as X. Normal distribution is parameterized by μ and σ. \[S = \frac{QK^\top}{\sqrt{d_k}} \in \mathbb{R}^{n \times m}\]
Symbols: \(Q \in \mathbb{R}^{n \times d_k}\) = query matrix; \(K \in \mathbb{R}^{m \times d_k}\) = key matrix; \(S_{ij} = \frac{q_i \cdot k_j}{\sqrt{d_k}}\) = scaled similarity score between query \(i\) and key \(j\).
Intuition: \(S_{ij}\) measures how compatible query \(i\) is with key \(j\); higher scores mean query \(i\) will incorporate more of value \(j\) in its output. \[\text{Explained variance ratio}_k = \frac{\sum_{i=1}^k \lambda_i}{\sum_{i=1}^d \lambda_i}\]
Symbols: \(\lambda_i\) = eigenvalues of the covariance matrix sorted descending; \(d\) = total features.
Intuition: The fraction of total variance captured by the top k components. Used to choose k: e.g., keep components explaining 95% of variance. \[\frac{\partial \mathcal{L}}{\partial z_i} = \hat{y}_i - y_i\]
Symbols: \(\hat{y}_i = \sigma(z_i)\) = softmax probability; \(y_i\) = true label (one-hot); \(z_i\) = logit.
Intuition: The gradient of cross-entropy + softmax w.r.t. logits is simply the prediction minus the label — a beautifully clean formula.

Question 23 AI Math (500 Questions)

What is the gradient of the loss w.r.t. weights in backpropagation?

\[A\mathbf{v} = \lambda\mathbf{v},\quad \mathbf{v} \neq \mathbf{0}\]
Symbols: \(A\) = square matrix; \(\lambda\) = eigenvalue (scalar); \(\mathbf{v}\) = eigenvector.
Intuition: An eigenvector is a special direction that A only stretches (by λ) but doesn't rotate. λ > 0: same direction; λ < 0: flipped; λ = 0: collapsed to zero. \[\frac{\partial \mathcal{L}}{\partial W^{(l)}} = \delta^{(l)} (a^{(l-1)})^\top\]
Symbols: \(\delta^{(l)}\) = error at layer l; \(a^{(l-1)}\) = activation from previous layer.
Intuition: Outer product of error signal and previous layer activation. Weight gradient = upstream gradient times pre-synaptic activation. This is why large activations amplify weight gradients. \[\theta_{t+1} = \theta_t - \eta \nabla_\theta \mathcal{L}(\theta_t)\]
Symbols: \(\eta\) = learning rate; \(\nabla_\theta \mathcal{L}\) = gradient of loss w.r.t. \(\theta\).
Intuition: Move parameters opposite to the gradient — downhill on the loss surface. Small η: safe but slow. Large η: fast but can diverge. \[\mathcal{L}(\theta_2) \geq \mathcal{L}(\theta_1) + \nabla\mathcal{L}(\theta_1)^\top(\theta_2-\theta_1) + \frac{m}{2}\|\theta_2-\theta_1\|^2\]
Symbols: \(m > 0\) = strong convexity constant; quadratic lower bound.
Intuition: Stronger than convexity — the function grows at least quadratically away from the minimum. Guarantees a unique global minimum and faster convergence (linear convergence rate for GD).

Question 24 AI Math (500 Questions)

What is the zeroth-order (constant) Taylor approximation?

\[f(\mathbf{x} + \boldsymbol{\delta}) \approx f(\mathbf{x})\]
Symbols: \(\boldsymbol{\delta}\) = perturbation (assumed tiny).
Intuition: Zero-order: just use the current value. Used in zero-order optimization (black-box optimization) where gradients are unavailable — only function evaluations. \[\nabla_{\mathbf{x}} (\mathbf{x}^\top A \mathbf{x}) = (A + A^\top)\mathbf{x}\]
Symbols: \(A \in \mathbb{R}^{n \times n}\); if A is symmetric: \(\nabla = 2A\mathbf{x}\).
Intuition: For the symmetric case (common in ML), gradient is 2Ax — analogous to the scalar derivative of ax² being 2ax. Key for understanding least squares. \[\mathcal{L}_{\mathrm{VAE}} = \mathbb{E}_{q_\phi}[\log p_\theta(x|z)] - D_{\mathrm{KL}}(q_\phi(z|x)\|p(z))\]
Symbols: \(q_\phi(z|x)\) = encoder; \(p_\theta(x|z)\) = decoder; \(p(z)=\mathcal{N}(0,I)\) = prior.
Intuition: Two terms: reconstruction loss (how well decoder reconstructs x) − KL regularization (how close encoder posterior is to prior). The KL term prevents encoder from ignoring the prior and ensures the latent space is smooth. \[\theta_{t+1} = \arg\min_\theta \langle\nabla f(\theta_t), \theta\rangle + \frac{1}{\eta}D_\phi(\theta,\theta_t)\]
Symbols: \(D_\phi\) = Bregman divergence; generalizes gradient descent to non-Euclidean geometry.
Intuition: Instead of Euclidean distance (GD), uses a Bregman divergence to measure 'closeness'. For the negative entropy mirror map → exponentiated gradient. Useful when θ lies on a simplex.

Question 25 AI Math (500 Questions)

How does the trace relate to eigenvalues?

\[\theta_{t+1} = \arg\min_\theta \langle\nabla f(\theta_t), \theta\rangle + \frac{1}{\eta}D_\phi(\theta,\theta_t)\]
Symbols: \(D_\phi\) = Bregman divergence; generalizes gradient descent to non-Euclidean geometry.
Intuition: Instead of Euclidean distance (GD), uses a Bregman divergence to measure 'closeness'. For the negative entropy mirror map → exponentiated gradient. Useful when θ lies on a simplex. \[\mathcal{L}(\theta_2) \geq \mathcal{L}(\theta_1) + \nabla\mathcal{L}(\theta_1)^\top(\theta_2-\theta_1) + \frac{m}{2}\|\theta_2-\theta_1\|^2\]
Symbols: \(m > 0\) = strong convexity constant; quadratic lower bound.
Intuition: Stronger than convexity — the function grows at least quadratically away from the minimum. Guarantees a unique global minimum and faster convergence (linear convergence rate for GD). \[\mathrm{tr}(A) = \sum_{i=1}^n \lambda_i\]
Symbols: \(\lambda_i\) = eigenvalues of A.
Intuition: The trace is the sum of all eigenvalues (counting multiplicity). Together with the determinant (product of eigenvalues), these are key matrix invariants. \[\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v}_t}+\epsilon}\hat{m}_t\]
Symbols: \(\hat{m}_t\) = bias-corrected 1st moment; \(\hat{v}_t\) = bias-corrected 2nd moment; \(\epsilon \approx 10^{-8}\).
Intuition: Step size ≈ η × (gradient direction) / (gradient scale). Approximately unit step size per parameter. Default hyperparameters (η=0.001, β₁=0.9, β₂=0.999) work well out-of-the-box for most tasks.

Question 26 AI Math (500 Questions)

What is the cross-entropy between the empirical data distribution and the model?

\[\mathcal{L}_{\mathrm{VAE}} = \mathbb{E}_{q_\phi}[\log p_\theta(x|z)] - D_{\mathrm{KL}}(q_\phi(z|x)\|p(z))\]
Symbols: \(q_\phi(z|x)\) = encoder; \(p_\theta(x|z)\) = decoder; \(p(z)=\mathcal{N}(0,I)\) = prior.
Intuition: Two terms: reconstruction loss (how well decoder reconstructs x) − KL regularization (how close encoder posterior is to prior). The KL term prevents encoder from ignoring the prior and ensures the latent space is smooth. \[\mathcal{L} = -\log\hat{y}_{\mathrm{true}}; \quad \lim_{\hat{y}\to 0}-\log\hat{y} = \infty\]
Symbols: \(\hat{y}_{\mathrm{true}}\) = predicted probability for correct class.
Intuition: Log blows up near zero. A confident wrong prediction (true class probability → 0) gives infinite loss. This strongly penalizes overconfident mistakes — more so than squared error. \[\mathrm{Dir}(\boldsymbol{\alpha}) \propto \prod_{k=1}^K p_k^{\alpha_k-1}, \quad \sum_k p_k = 1\]
Symbols: \(\boldsymbol{\alpha}\) = concentration parameters; \(p_k\) = probability values.
Intuition: Distribution over probability vectors (simplex). Conjugate prior for the Categorical/Multinomial. Large α concentrates mass near the mean; small α spreads mass to corners. \[H(p_{\mathrm{data}},p_\theta) = -\frac{1}{N}\sum_{i=1}^N \log p_\theta(x_i)\]
Symbols: \(p_{\mathrm{data}}\) = uniform over training set; \(p_\theta\) = model.
Intuition: The average negative log-likelihood — the standard training loss. Minimizing this is exactly MLE and exactly minimizing KL from data distribution to model.

Question 27 AI Math (500 Questions)

Why is KL divergence always non-negative?

\[\min_{p(z|x)} I(X;Z) - \beta I(Z;Y)\]
Symbols: \(Z\) = compressed representation; \(Y\) = target; \(\beta\) = tradeoff; minimize compression while preserving prediction.
Intuition: Find the minimal sufficient statistic for predicting Y from X. The tradeoff parameter β controls compression vs. accuracy. Theoretical lens on what representations DNNs learn. \[D_{\mathrm{KL}}(P\|Q) \geq 0, \text{ with equality iff } P=Q\]
Symbols: Follows from Jensen's inequality: \(-\log\) is convex.
Intuition: By Jensen's: \(\mathbb{E}_P[-\log(Q/P)] \geq -\log\mathbb{E}_P[Q/P] = -\log 1 = 0\). Extra code length is always non-negative — you can't do better than the true distribution. \[A = A^\top \implies \lambda_i \in \mathbb{R} \text{ and eigenvectors are orthogonal}\]
Symbols: \(A^\top\) = transpose; \(\lambda_i\) = eigenvalues.
Intuition: The spectral theorem: symmetric matrices are always diagonalizable with real eigenvalues and mutually orthogonal eigenvectors. Covariance matrices and Hessians benefit from this. \[P(|X - \mu| \geq k\sigma) \leq \frac{1}{k^2}\]
Symbols: \(\mu = \mathbb{E}[X]\); \(\sigma^2 = \mathrm{Var}(X)\); \(k > 0\).
Intuition: At least 1-1/k² of the probability mass lies within k standard deviations of the mean, for any distribution with finite variance. Weaker than Gaussian but universally applicable.

Question 28 AI Math (500 Questions)

What is matrix multiplication and when is it defined?

\[S = \frac{QK^\top}{\sqrt{d_k}} \in \mathbb{R}^{n \times m}\]
Symbols: \(Q \in \mathbb{R}^{n \times d_k}\) = query matrix; \(K \in \mathbb{R}^{m \times d_k}\) = key matrix; \(S_{ij} = \frac{q_i \cdot k_j}{\sqrt{d_k}}\) = scaled similarity score between query \(i\) and key \(j\).
Intuition: \(S_{ij}\) measures how compatible query \(i\) is with key \(j\); higher scores mean query \(i\) will incorporate more of value \(j\) in its output. \[\hat{x}_{n,c,h,w} = \frac{x_{n,c,h,w} - \mu_{n,c}}{\sqrt{\sigma_{n,c}^2+\epsilon}},\quad \mu_{n,c} = \frac{1}{HW}\sum_{h,w} x_{n,c,h,w}\]
Symbols: Normalizes each sample's each channel independently over spatial dimensions H,W.
Intuition: Per-instance, per-channel normalization. Removes instance-specific contrast information. Used in neural style transfer — normalizing style statistics enables better style transfer. \[\text{GELU}(x) = x\,\Phi(x) = x \cdot \frac{1}{2}\left[1+\mathrm{erf}\!\left(\frac{x}{\sqrt{2}}\right)\right]\]
Symbols: \(\Phi(x)\) = standard normal CDF; \(\mathrm{erf}\) = error function.
Intuition: Weights input by its probability of being greater than 0 under a standard Gaussian. Smooth everywhere (unlike ReLU). Widely used in transformers (BERT, GPT). Can be approximated as 0.5x(1+tanh(√(2/π)(x+0.044715x³))). \[C = AB,\quad C_{ij} = \sum_{k=1}^p A_{ik} B_{kj}\]
Symbols: \(A \in \mathbb{R}^{m \times p}\); \(B \in \mathbb{R}^{p \times n}\); \(C \in \mathbb{R}^{m \times n}\).
Intuition: Each entry C_ij is the dot product of row i of A with column j of B. Represents composing two linear transformations.

Question 29 AI Math (500 Questions)

What are the basic differentiation rules for power and constant functions?

\[\frac{d}{dx}[x^n] = nx^{n-1}, \quad \frac{d}{dx}[c] = 0\]
Symbols: \(n\) = exponent; \(c\) = constant.
Intuition: A constant has zero slope. A power function: drop the exponent as a coefficient, reduce exponent by 1. Foundation for polynomial differentiation. \[A = A^\top \iff A_{ij} = A_{ji} \text{ for all } i,j\]
Symbols: \(A^\top\) = transpose of A.
Intuition: A symmetric matrix equals its own transpose. Covariance matrices and Hessians are symmetric. Symmetric matrices always have real eigenvalues. \[p\text{-value} = P(\text{test statistic} \geq t_{\text{obs}} | H_0)\]
Symbols: \(H_0\) = null hypothesis; \(t_{\text{obs}}\) = observed test statistic.
Intuition: Probability of observing results at least as extreme as yours, assuming H0 is true. Small p-value → data unlikely under H0 → evidence against H0. Not the probability H0 is true. \[\nabla f(\mathbf{x}^*) = \mathbf{0}\]
Symbols: \(\mathbf{x}^*\) = critical point; all partial derivatives are zero.
Intuition: A point where the gradient vanishes — could be a minimum, maximum, or saddle point. Gradient descent converges to critical points but not necessarily to global minima.

Question 30 AI Math (500 Questions)

What is the exponential family of distributions?

\[D_{\mathrm{KL}}(P\|Q) = \sum_x p(x)\log\frac{p(x)}{q(x)}\]
Symbols: \(P\) = true distribution; \(Q\) = approximate distribution; result in nats or bits.
Intuition: Extra bits needed to code samples from P using a code optimized for Q. Measures information lost by approximating P with Q. \[p(x|\eta) = h(x)\exp(\eta^\top T(x) - A(\eta))\]
Symbols: \(\eta\) = natural parameter; \(T(x)\) = sufficient statistic; \(A(\eta)\) = log-partition function.
Intuition: A broad family including Gaussian, Bernoulli, Poisson, Gamma. Has conjugate priors, sufficient statistics, and convenient optimization properties. \[\mathrm{Var}(X) = \mathbb{E}[\mathrm{Var}(X|Y)] + \mathrm{Var}(\mathbb{E}[X|Y])\]
Symbols: First term = average within-group variance; second = between-group variance.
Intuition: Total variance splits into variance explained by Y (the between term) and unexplained variance (within term). Used in Rao-Blackwell theorem and random effects models. \[\mathcal{I}(\theta) = \mathbb{E}\left[\nabla_\theta \log p(x|\theta) \, (\nabla_\theta \log p(x|\theta))^\top\right]\]
Symbols: \(\mathcal{I}(\theta)\) = Fisher information; outer product of score vectors.
Intuition: Measures how informative data is about θ. Used in natural gradient descent, the Cramér-Rao bound (variance lower bound for estimators), and second-order optimization.

Question 31 AI Math (500 Questions)

What is cross-entropy?

\[H(P,Q) = -\sum_x p(x)\log q(x)\]
Symbols: \(P\) = true distribution; \(Q\) = model distribution; result = expected code length under Q.
Intuition: Average number of bits needed to encode events from P using code optimized for Q. Always ≥ H(P). Minimizing cross-entropy is the standard classification training objective. \[S = \frac{QK^\top}{\sqrt{d_k}} \in \mathbb{R}^{n \times m}\]
Symbols: \(Q \in \mathbb{R}^{n \times d_k}\) = query matrix; \(K \in \mathbb{R}^{m \times d_k}\) = key matrix; \(S_{ij} = \frac{q_i \cdot k_j}{\sqrt{d_k}}\) = scaled similarity score between query \(i\) and key \(j\).
Intuition: \(S_{ij}\) measures how compatible query \(i\) is with key \(j\); higher scores mean query \(i\) will incorporate more of value \(j\) in its output. \[Q^{-1} = Q^\top \iff Q^\top Q = I\]
Symbols: \(Q\) = orthogonal matrix; \(I\) = identity.
Intuition: For orthogonal matrices, the inverse is simply the transpose — cheap to compute. These represent rotations/reflections that preserve lengths and angles. \[\text{PE}(\text{pos},\, 2i+1) = \cos\!\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)\]
Symbols: \(\text{pos}\) = sequence position; \(i\) = dimension pair index; cosine is paired with sine at the same frequency to form a 2D rotation representation.
Intuition: Pairing sin and cos at each frequency means the encoding for \(\text{pos}+k\) can be expressed as a linear (rotation) transformation of the encoding for \(\text{pos}\), letting the model generalise to relative offsets.

Question 32 AI Math (500 Questions)

What is the Poisson distribution?

\[\mathcal{L}_\delta(r) = \begin{cases}\frac{1}{2}r^2 & |r| \leq \delta \\ \delta|r| - \frac{1}{2}\delta^2 & |r| > \delta\end{cases}\]
Symbols: \(r = y - \hat{y}\) = residual; \(\delta\) = threshold between quadratic and linear regions.
Intuition: Quadratic for small residuals (smooth, fast convergence), linear for large residuals (outlier-robust). Best of MSE and MAE. Used in regression with outliers (DQN Q-value loss). \[|\theta| = d_{\text{in}} \times d_{\text{out}} + d_{\text{out}}\]
Symbols: \(d_{\text{in}} \times d_{\text{out}}\) = weight matrix entries; \(d_{\text{out}}\) = bias terms.
Intuition: Each output unit connects to all input units (weight) plus has a bias. A 768→768 layer has 768²+768 ≈ 590K parameters. Deep networks have millions/billions total. \[\text{FC: } d_{\text{in}}\times d_{\text{out}}; \quad \text{Conv: } k^2 \times C_{\text{in}} \times C_{\text{out}}\]
Symbols: \(k\) = kernel size; \(C_{\text{in}}, C_{\text{out}}\) = input/output channels; independent of spatial size.
Intuition: A conv layer with k=3, 64 channels in/out has 3²×64×64=36,864 params — vs. a FC layer connecting 64×H×W input to 64×H×W output which has (64HW)² params. Enormous savings via weight sharing. \[P(X=k) = \frac{\lambda^k e^{-\lambda}}{k!}, \quad k \in \{0,1,2,\ldots\}, \quad \mathbb{E}[X] = \mathrm{Var}(X) = \lambda\]
Symbols: \(\lambda > 0\) = average rate; \(k!\) = factorial.
Intuition: Models count of rare events in a fixed interval (arrivals, word counts). Unique property: mean equals variance. Widely used in NLP language models.

Question 33 AI Math (500 Questions)

What is the Frobenius norm of a matrix?

\[\nabla f(\mathbf{x}) = \begin{bmatrix}\frac{\partial f}{\partial x_1} \\ \vdots \\ \frac{\partial f}{\partial x_n}\end{bmatrix}\]
Symbols: \(\nabla\) = nabla (gradient operator); \(\partial f / \partial x_i\) = i-th partial derivative.
Intuition: The vector of all partial derivatives. Points in the direction of steepest ascent of f. Its magnitude is the steepness. Used directly in gradient descent. \[\frac{d}{dx}\sigma(x) = \sigma(x)(1-\sigma(x))\]
Symbols: \(\sigma(x) = 1/(1+e^{-x})\) = sigmoid function.
Intuition: Elegant self-referential formula. Maximum at x=0 (value 0.25). Vanishes for large |x| (saturation) — this causes vanishing gradients in deep sigmoid networks. \[F_l \approx A_{l-1} \otimes G_l, \quad A_{l-1} = \mathbb{E}[aa^\top],\; G_l = \mathbb{E}[gg^\top]\]
Symbols: \(A_{l-1}\) = activation covariance; \(G_l\) = gradient covariance; \(\otimes\) = Kronecker product.
Intuition: Approximates the Fisher information matrix as a Kronecker product of two smaller matrices per layer. Makes natural gradient tractable for large networks. Used to train networks faster with better conditioning. \[\|A\|_F = \sqrt{\sum_{i,j} A_{ij}^2}\]
Symbols: \(A_{ij}\) = element at row i, column j.
Intuition: The L2 norm applied to a matrix by treating it as a flattened vector. Equivalent to \(\sqrt{\mathrm{tr}(A^\top A)}\).

Question 34 AI Math (500 Questions)

What is the determinant of the Jacobian (Jacobian determinant) used for?

\[\left|\det(J)\right| = \text{volume scaling factor of the transformation } \mathbf{f}\]
Symbols: \(J = \partial\mathbf{f}/\partial\mathbf{x}\); \(|\det J|\) = absolute Jacobian determinant.
Intuition: Appears in change-of-variable formulas for probability densities: \(p_x(\mathbf{x}) = p_z(\mathbf{f}(\mathbf{x}))|\det J|\). Central in normalizing flows. \[\theta_{t+1} = \theta_t - \eta_t \nabla_\theta \ell(f_\theta(x_i), y_i)\]
Symbols: \((x_i, y_i)\) = single randomly sampled training example; noisy gradient estimate.
Intuition: Instead of computing the full gradient over all N examples, use a single random example. Much cheaper per step (O(1) vs O(N)), converges in expectation to a minimum. \[Q = XW_Q,\quad K = XW_K,\quad V = XW_V\]
Symbols: \(W_Q, W_K \in \mathbb{R}^{d_{\text{model}} \times d_k}\); \(W_V \in \mathbb{R}^{d_{\text{model}} \times d_v}\); all three are learned parameters updated during training.
Intuition: The three projections decompose each token's representation into a query (what to look for), a key (what it contains), and a value (what to output), enabling flexible content-based retrieval. \[P(\theta|x_1,\ldots,x_n) \propto \left(\prod_{i=1}^n P(x_i|\theta)\right) P(\theta)\]
Symbols: i.i.d. observations; product of likelihoods × prior.
Intuition: With independent observations, likelihoods multiply. In log space: log posterior = Σ log likelihood + log prior. MAP finds the peak of this.

Question 35 AI Math (500 Questions)

What is the derivative of tanh?

\[\text{ReLU}(x) = \max(0,x) = \begin{cases}x & x>0 \\ 0 & x\leq 0\end{cases}\]
Symbols: Rectified linear unit; derivative = 1 for x>0, 0 for x<0.
Intuition: Simple and effective: no saturation for positive inputs (gradient = 1, no vanishing). Fast to compute. Problem: dying ReLU — if z is always ≤0, neuron gets no gradient and 'dies'. Default choice for hidden layers. \[\mathcal{L}_{\mathrm{MAE}} = \frac{1}{N}\sum_{i=1}^N |y_i - \hat{y}_i|\]
Symbols: Absolute value instead of square; gradient = sign(ŷ-y).
Intuition: More robust to outliers than MSE (linear not quadratic penalty). But non-differentiable at 0, and gradient doesn't decrease near minimum. Huber loss (smooth L1) combines best of both. \[\frac{d}{dx}\tanh(x) = 1 - \tanh^2(x) = \mathrm{sech}^2(x)\]
Symbols: \(\tanh(x) = (e^x-e^{-x})/(e^x+e^{-x})\); \(\mathrm{sech}\) = hyperbolic secant.
Intuition: Max derivative = 1 at x=0; vanishes near saturation. Better than sigmoid (zero-centered output) but still saturates. Used in LSTMs and RNNs. \[f(\lambda\mathbf{x} + (1-\lambda)\mathbf{y}) \leq \lambda f(\mathbf{x}) + (1-\lambda)f(\mathbf{y}), \quad \lambda \in [0,1]\]
Symbols: \(\lambda\) = interpolation parameter; \(\mathbf{x}, \mathbf{y}\) = any two points.
Intuition: The chord between any two points lies above the function. Convex functions have no local minima that aren't global — gradient descent is guaranteed to converge.

Question 36 AI Math (500 Questions)

What is the entropy rate of a stationary process?

\[\mathcal{H} = \lim_{N\to\infty} \frac{1}{N}H(X_1,\ldots,X_N)\]
Symbols: \(\mathcal{H}\) = entropy rate; limit of average entropy per symbol.
Intuition: How much new information each new symbol adds, on average, in a long sequence. Language has entropy rate ≈ 1 bit/character (Shannon's experiment). Language models are optimized to minimize this. \[A = Q\Lambda Q^{-1}\]
Symbols: \(Q\) = matrix of eigenvectors (columns); \(\Lambda = \mathrm{diag}(\lambda_1,\ldots,\lambda_n)\) = diagonal eigenvalue matrix.
Intuition: Factorizes A into: rotate to eigenbasis (Q^-1), scale each axis by eigenvalue (Λ), rotate back (Q). Possible only if A has n linearly independent eigenvectors. \[\text{Self: } Q,K,V \leftarrow X;\quad \text{Cross: } Q \leftarrow Y,\; K,V \leftarrow X\]
Symbols: \(X\) = source sequence (e.g., encoder output); \(Y\) = target sequence (e.g., partial decoder output); arrows show which sequence provides each projection.
Intuition: Self-attention gathers context within one sequence; cross-attention lets one sequence (the decoder) attend to a different sequence (the encoder), enabling alignment between source and target. \[\mathrm{proj}_{\mathbf{u}}\mathbf{v} = \frac{\mathbf{v} \cdot \mathbf{u}}{\|\mathbf{u}\|^2}\mathbf{u}\]
Symbols: \(\mathbf{u}\) = direction to project onto; \(\mathbf{v} \cdot \mathbf{u}\) = dot product.
Intuition: The 'shadow' of v onto u. The scalar coefficient gives how much of v lies in the direction of u.

Question 37 AI Math (500 Questions)

What is the Gaussian (Normal) distribution?

\[\mathcal{N}(x|\mu,\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)\]
Symbols: \(\mu\) = mean; \(\sigma^2\) = variance; \(\sigma\) = standard deviation; \(x \in (-\infty,\infty)\).
Intuition: The bell curve. Most important distribution: arises from CLT, maximum entropy for fixed mean/variance, closed under linear operations. Cornerstone of ML. \[p(x, y) = P(X=x, Y=y)\]
Symbols: \(p(x,y)\) = probability of both X=x and Y=y occurring.
Intuition: Describes the combined behavior of two (or more) variables simultaneously. All marginals and conditionals can be derived from the joint. \[\eta_t = \eta_{\min} + \frac{1}{2}(\eta_{\max}-\eta_{\min})\left(1 + \cos\frac{\pi t}{T}\right)\]
Symbols: \(T\) = total steps; \(\eta_{\max}, \eta_{\min}\) = max and min learning rates.
Intuition: Smoothly decays the learning rate following a cosine curve. Starts high (fast progress), ends near zero (fine-tuning). Widely used with transformer training. \[\hat{\mathbf{x}} = (A^\top A)^{-1}A^\top \mathbf{b}\]
Symbols: \(A^\top A\) = Gram matrix; \(A^\top \mathbf{b}\) = normal equations right-hand side.
Intuition: Solves Ax ≈ b when no exact solution exists, by projecting b onto col(A). The normal equations \(A^\top A\hat{x} = A^\top b\) characterize the solution.

Question 38 AI Math (500 Questions)

What is the gradient of a linear function f(x) = wᵀx?

\[\nabla_{\mathbf{x}} (\mathbf{w}^\top\mathbf{x}) = \mathbf{w}\]
Symbols: \(\mathbf{w}, \mathbf{x} \in \mathbb{R}^n\).
Intuition: The gradient of a linear function is just its coefficient vector. Constant gradient means constant slope in all directions — a flat (hyperplane) landscape. \[\hat{\mathbf{x}} = A^+\mathbf{b} = \arg\min_{\mathbf{x}} \|A\mathbf{x}-\mathbf{b}\|_2 \text{ with min } \|\mathbf{x}\|_2\]
Symbols: \(A^+\) = pseudoinverse; \(\mathbf{b}\) = target vector.
Intuition: Among all solutions that minimize the residual ‖Ax-b‖, picks the one with smallest norm. Key in underdetermined systems (more unknowns than equations). \[\text{PE}(\text{pos},\, 2i) = \sin\!\left(\frac{\text{pos}}{10000^{2i/d_{\text{model}}}}\right)\]
Symbols: \(\text{pos}\) = token position (0-indexed); \(i\) = dimension pair index; \(d_{\text{model}}\) = embedding size; \(10000^{2i/d_{\text{model}}}\) = wavelength scale factor forming a geometric progression.
Intuition: Each dimension encodes position with a sine wave at a different frequency; small \(i\) gives fast oscillation (local structure), large \(i\) gives slow oscillation (global structure), together uniquely encoding every position. \[\mathcal{L}(\theta_T) - \mathcal{L}(\theta^*) \leq \frac{L\|\theta_0 - \theta^*\|^2}{2T}\]
Symbols: \(T\) = iterations; \(\theta^*\) = optimal parameters; \(L\) = smoothness.
Intuition: The loss excess decreases as O(1/T) — sublinear convergence. For strongly convex functions it's O(ρ^T) (linear/geometric convergence), which is much faster.

Question 39 AI Math (500 Questions)

What is Flash Attention and why is it faster than standard attention?

\[\nabla_\theta \mathbb{E}_{p_\theta}[f(x)] = \mathbb{E}_{p_\theta}\left[f(x)\nabla_\theta \log p_\theta(x)\right]\]
Symbols: \(\nabla_\theta \log p_\theta(x)\) = score function; \(f(x)\) = reward/objective.
Intuition: Uses the log-derivative trick to compute gradient of expectation. Used in REINFORCE (RL) and when reparameterization is not possible (discrete latents). \[\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V \quad\text{computed without materialising }S \in \mathbb{R}^{n \times n}\]
Symbols: \(S = QK^\top/\sqrt{d_k}\) = full score matrix (\(n^2\) entries); Flash Attention tiles Q, K, V into SRAM blocks and fuses score computation, softmax, and value aggregation into one GPU kernel without writing \(S\) to HBM.
Intuition: Standard attention reads/writes the \(n \times n\) matrix to slow GPU HBM; Flash Attention keeps live data in fast on-chip SRAM by processing tiles, reducing HBM traffic from \(\mathcal{O}(n^2)\) to \(\mathcal{O}(n)\) and giving large wall-clock speedups on modern hardware. \[\hat{Y} = \sigma^{(L)}\!\left(W^{(L)}\sigma^{(L-1)}\!\left(\cdots\sigma^{(1)}\!\left(W^{(1)}X^\top + \mathbf{b}^{(1)}\right)\right) + \mathbf{b}^{(L)}\right)\]
Symbols: \(X \in \mathbb{R}^{N\times d}\) = batch; \(\hat{Y} \in \mathbb{R}^{N\times K}\) = predictions; L = depth.
Intuition: The complete forward pass stacks L layers of linear + activation. Non-linearities σ are necessary — without them, composition of linear maps is still linear. \[\theta_{t+1} = \theta_t - \eta\left(\frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon} + \lambda\theta_t\right)\]
Symbols: \(\lambda\) = weight decay coefficient; weight decay applied directly, not through the gradient.
Intuition: Adam + decoupled weight decay. In original Adam, L2 reg is divided by √v̂ (adaptive scaling), which weakens its effect. AdamW applies weight decay directly — better generalization. Default in modern transformer training.

Question 40 AI Math (500 Questions)

What is the evidence lower bound (ELBO) in terms of reconstruction and KL?

\[\mathcal{L} = \mathbb{E}_{q_\phi}[\log p_\theta(x|z)] - D_{\mathrm{KL}}(q_\phi(z|x)\|p(z)) \leq \log p_\theta(x)\]
Symbols: \(\leq \log p_\theta(x)\) = ELBO bounds the marginal log-likelihood from below.
Intuition: Maximizing ELBO is equivalent to simultaneously maximizing reconstruction likelihood and minimizing KL to prior. The gap between ELBO and log p(x) equals KL(q||p(z|x)) — the quality of the variational approximation. \[\mathbf{a} + \mathbf{b} = \begin{bmatrix}a_1+b_1 \\ a_2+b_2 \\ \vdots \\ a_n+b_n\end{bmatrix}\]
Symbols: \(a_i, b_i\) = corresponding components of vectors a and b.
Intuition: Add element-wise. Geometrically, place the tail of b at the head of a; the sum points to the new head. \[\mathbf{z} = \mu + \sigma \odot \boldsymbol{\epsilon}, \quad \boldsymbol{\epsilon} \sim \mathcal{N}(0, I)\]
Symbols: \(\mu, \sigma\) = learned mean/std; \(\odot\) = element-wise multiply; \(\boldsymbol{\epsilon}\) = noise sample.
Intuition: Makes the sampling operation differentiable by moving the stochasticity to ε (which doesn't depend on parameters). Enables backprop through stochastic nodes in VAEs. \[S_{ij}^{\text{masked}} = \begin{cases} S_{ij} & j \le i \\ -\infty & j > i \end{cases}\]
Symbols: \(S_{ij}\) = raw score from position \(i\) to \(j\); setting future positions to \(-\infty\) forces their softmax weight to 0; \(j > i\) = future positions.
Intuition: Autoregressive models must not use future tokens during training or inference; masking future positions with \(-\infty\) before softmax enforces that position \(i\) only attends to positions \(\le i\).

Question 41 AI Math (500 Questions)

What is the minimum-norm least-squares solution using the pseudoinverse?

\[\hat{\mathbf{x}} = A^+\mathbf{b} = \arg\min_{\mathbf{x}} \|A\mathbf{x}-\mathbf{b}\|_2 \text{ with min } \|\mathbf{x}\|_2\]
Symbols: \(A^+\) = pseudoinverse; \(\mathbf{b}\) = target vector.
Intuition: Among all solutions that minimize the residual ‖Ax-b‖, picks the one with smallest norm. Key in underdetermined systems (more unknowns than equations). \[\eta_{\mathrm{eff}} \approx \eta \frac{\hat{m}_t}{\sqrt{\hat{v}_t}} \approx \eta \cdot \text{sign}(g_t) \text{ (heuristically)}\]
Symbols: \(\hat{m}_t/\sqrt{\hat{v}_t}\) ≈ signal-to-noise ratio of gradient; bounded by ±1.
Intuition: Adam normalizes gradient by its own scale — approximately equivalent to taking unit steps in the gradient direction. This makes it robust to gradient scale but can hurt generalization compared to SGD. \[\text{Local: } \mathcal{L}(\theta^*) \leq \mathcal{L}(\theta) \text{ for } \|\theta-\theta^*\|<\delta; \quad \text{Global: for all } \theta\]
Symbols: \(\delta\) = small neighborhood radius.
Intuition: A local minimum is better than all nearby points but might not be the best overall. Deep nets have many local minima, but in practice most are near-equivalent (flat minima). Global min guarantees exist only for convex problems. \[\mathcal{N}(\mathbf{x}|\boldsymbol{\mu}, \mathrm{diag}(\boldsymbol{\sigma}^2)) = \prod_{i=1}^d \mathcal{N}(x_i|\mu_i,\sigma_i^2)\]
Symbols: \(\boldsymbol{\sigma}^2\) = vector of per-dimension variances; factorizes into independent 1D Gaussians.
Intuition: Diagonal covariance = independent features. The joint density factorizes into a product. This is the assumption in many variational autoencoders and Gaussian Naïve Bayes.

Question 42 AI Math (500 Questions)

What is the soft-thresholding operator for L1 regularization?

\[\mathrm{prox}_{\eta\lambda\|\cdot\|_1}(\theta_j) = \mathrm{sign}(\theta_j)\max(|\theta_j|-\eta\lambda, 0)\]
Symbols: \(\lambda\) = L1 penalty; \(\eta\) = step size; shrinks small weights to exactly zero.
Intuition: The proximal operator for L1 shrinks each weight toward zero by η λ, setting it to zero if it would overshoot. This is what makes L1 regularization produce sparse solutions. \[\mathrm{Var}[\hat{g}] = \frac{\mathrm{Var}[g_i]}{B}\]
Symbols: \(B\) = batch size; \(g_i\) = single-sample gradient; variance decreases as \(1/B\).
Intuition: Averaging B gradients reduces variance by a factor of B. Doubling batch size halves gradient variance but also doubles per-step cost. You can scale LR ∝ √B (or ∝ B with linear scaling rule). \[\sigma_g^2 = \frac{S}{\text{batch size}}, \quad S = \frac{1}{N}\sum_i \|\nabla\ell_i - \nabla\mathcal{L}\|^2\]
Symbols: \(S\) = gradient noise scale; \(N\) = dataset size; \(\nabla\ell_i\) = per-sample gradient.
Intuition: Gradient noise scale S measures the signal-to-noise ratio of mini-batch gradients. OpenAI found that critical batch size ≈ S determines when batch size stops helping — theory for LLM training budget allocation. \[A = LL^\top, \quad L \text{ lower triangular with positive diagonal}\]
Symbols: \(L\) = lower-triangular Cholesky factor.
Intuition: A symmetric PSD matrix can be factored as LL^T. Used to efficiently solve linear systems and sample from multivariate Gaussians. Fails if A is not PSD.

Question 43 AI Math (500 Questions)

What is the softmax activation function?

\[M_X(t) = \mathbb{E}[e^{tX}]\]
Symbols: \(t \in \mathbb{R}\); k-th moment: \(\mathbb{E}[X^k] = M_X^{(k)}(0)\) (k-th derivative at 0).
Intuition: A function that encodes all moments. Differentiate k times and evaluate at t=0 to get E[X^k]. MGF uniquely characterizes a distribution (when it exists). \[\Sigma = \mathbb{E}[(\mathbf{x}-\boldsymbol{\mu})(\mathbf{x}-\boldsymbol{\mu})^\top]\]
Symbols: \(\boldsymbol{\mu} = \mathbb{E}[\mathbf{x}]\) = mean vector; \(\Sigma_{ij} = \mathrm{Cov}(x_i, x_j)\).
Intuition: Encodes the variance of each feature (diagonal) and co-movement between features (off-diagonal). Always symmetric and positive semi-definite. \[\text{softmax}(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^K e^{z_j}}\]
Symbols: \(\mathbf{z} \in \mathbb{R}^K\) = logits; output is a probability distribution over K classes; \(\sum_i \text{softmax}(\mathbf{z})_i = 1\).
Intuition: Converts logits to probabilities. Differentiable approximation of argmax. Temperature-scaled: softmax(z/T) → uniform as T→∞, one-hot as T→0. \[\mathrm{prox}_{\eta\lambda\|\cdot\|_1}(\theta_j) = \mathrm{sign}(\theta_j)\max(|\theta_j|-\eta\lambda, 0)\]
Symbols: \(\lambda\) = L1 penalty; \(\eta\) = step size; shrinks small weights to exactly zero.
Intuition: The proximal operator for L1 shrinks each weight toward zero by η λ, setting it to zero if it would overshoot. This is what makes L1 regularization produce sparse solutions.

Question 44 AI Math (500 Questions)

What is the receptive field in a CNN?

\[\theta_j^{t+1} = \arg\min_{\theta_j} \mathcal{L}(\theta_1^{t+1},\ldots,\theta_{j-1}^{t+1},\theta_j,\theta_{j+1}^t,\ldots)\]
Symbols: Minimize over one coordinate \(\theta_j\) at a time, holding others fixed.
Intuition: Each step optimizes a 1D subproblem. Useful when each coordinate has a closed-form solution (e.g., LASSO via coordinate descent). Efficient for large sparse problems. \[\theta_{t+1} = \theta_t - [H(\theta_t)]^{-1}\nabla\mathcal{L}(\theta_t)\]
Symbols: \(H = \nabla^2\mathcal{L}\) = Hessian; \(H^{-1}\) = inverse Hessian.
Intuition: Uses curvature (Hessian) to scale the gradient — takes a large step in flat directions and small step in steep directions. Quadratic convergence near the optimum. Too expensive for DL (O(n²) memory, O(n³) inversion). \[f(\mathbf{x} + \boldsymbol{\delta}) \approx f(\mathbf{x}) + \nabla f^\top\boldsymbol{\delta} + \frac{1}{2}\boldsymbol{\delta}^\top H\boldsymbol{\delta}\]
Symbols: \(H\) = Hessian matrix; \(\boldsymbol{\delta}\) = perturbation.
Intuition: Adds a curvature correction term ½δᵀHδ. Positive definite H means this term resists large steps. Basis for Newton's method and second-order optimizers. \[\text{Receptive field at layer } l = k + (k-1)(l-1) \text{ (for kernel size k)}\]
Symbols: \(k\) = kernel size; \(l\) = depth; grows linearly with depth for standard convolution.
Intuition: How large a region of the input influences one output neuron. Deeper layers see a larger context. Dilated convolutions grow the receptive field exponentially (used in WaveNet, DeepLab).

Question 45 AI Math (500 Questions)

What is a 2D convolution in CNNs?

\[(I * K)[i,j] = \sum_{m}\sum_{n} I[i+m,\,j+n]\,K[m,n]\]
Symbols: \(I\) = input feature map; \(K\) = kernel; \((i,j)\) = output spatial position.
Intuition: Slides a 2D filter over a 2D input. Each output pixel = weighted sum of a local patch. Strong inductive bias: local connectivity + translation equivariance. Huge parameter efficiency vs. fully connected. \[\text{if } \|\mathbf{g}\| > \tau: \quad \mathbf{g} \leftarrow \frac{\tau}{\|\mathbf{g}\|}\mathbf{g}\]
Symbols: \(\mathbf{g}\) = gradient vector; \(\tau\) = clipping threshold; \(\|\mathbf{g}\|\) = gradient norm.
Intuition: When gradients explode (huge norms), scale them back to have norm τ while preserving direction. Essential for training RNNs and transformers on long sequences. \[H_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j}\]
Symbols: \(H \in \mathbb{R}^{n \times n}\) for \(f: \mathbb{R}^n \to \mathbb{R}\); \(H = \nabla^2 f\).
Intuition: Matrix of all second partial derivatives. Captures the curvature of f. Symmetric (by Schwarz theorem). In optimization: tells whether a critical point is a min, max, or saddle. \[P(X=k) = p_k, \quad \sum_{k=1}^K p_k = 1\]
Symbols: \(K\) = number of categories; \(p_k \geq 0\) = probability of category k.
Intuition: Generalization of Bernoulli to K outcomes. Parameterized by a probability vector. Softmax output represents a Categorical distribution over class labels.

Question 46 AI Math (500 Questions)

How does PCA relate to SVD?

\[X = U\Sigma V^\top \implies \text{PCs} = V, \quad \Sigma^2/(N-1) = \Lambda\]
Symbols: \(X \in \mathbb{R}^{N \times d}\) = centered data matrix; \(V\) = right singular vectors = principal components.
Intuition: PCA via SVD of the centered data matrix is numerically more stable than eigendecomposing the covariance matrix, especially for high-dimensional data. \[p(x|y) = \frac{p(x,y)}{p(y)}\]
Symbols: \(p(y) = \sum_x p(x,y)\) = marginal of Y.
Intuition: Slice through the joint at a fixed y, then normalize. The conditional distribution is the joint normalized to integrate to 1 over x. \[F_l \approx A_{l-1} \otimes G_l, \quad A_{l-1} = \mathbb{E}[aa^\top],\; G_l = \mathbb{E}[gg^\top]\]
Symbols: \(A_{l-1}\) = activation covariance; \(G_l\) = gradient covariance; \(\otimes\) = Kronecker product.
Intuition: Approximates the Fisher information matrix as a Kronecker product of two smaller matrices per layer. Makes natural gradient tractable for large networks. Used to train networks faster with better conditioning. \[Z_i = \sum_j A_{ij}\, v_j,\quad A_{ij} = \text{softmax}\!\left(\frac{q_i \cdot k_j}{\sqrt{d_k}}\right)\]
Symbols: \(A_{ij}\) = input-dependent weight (recomputed per input, not a fixed parameter); \(v_j\) = value vector of token \(j\).
Intuition: A linear layer uses fixed learned weights; self-attention weights are computed on-the-fly from the input content, so the mixing pattern adapts dynamically to each input sequence.

Question 47 AI Math (500 Questions)

What is the mirror descent algorithm?

\[\theta_{t+1} = \arg\min_\theta \langle\nabla f(\theta_t), \theta\rangle + \frac{1}{\eta}D_\phi(\theta,\theta_t)\]
Symbols: \(D_\phi\) = Bregman divergence; generalizes gradient descent to non-Euclidean geometry.
Intuition: Instead of Euclidean distance (GD), uses a Bregman divergence to measure 'closeness'. For the negative entropy mirror map → exponentiated gradient. Useful when θ lies on a simplex. \[f(\mathbf{x}) = \mathbf{x}^\top A \mathbf{x} = \sum_{i,j} A_{ij} x_i x_j\]
Symbols: \(A \in \mathbb{R}^{n \times n}\) = symmetric matrix; \(\mathbf{x} \in \mathbb{R}^n\).
Intuition: A scalar function of a vector that is degree-2 in the components of x. The Hessian appears as A in the second-order Taylor expansion. A>0 means bowl-shaped. \[\mathrm{NMI}(X,Y) = \frac{I(X;Y)}{\sqrt{H(X)H(Y)}} \in [0,1]\]
Symbols: \(I(X;Y)\) = mutual information; normalized to [0,1] by geometric mean of entropies.
Intuition: Normalized version of MI suitable for comparing different pairs of variables. 0 = independent; 1 = perfectly dependent. Used to evaluate clustering quality (compare cluster labels to true labels). \[\frac{\partial}{\partial A}\log\det(A) = A^{-\top} = (A^{-1})^\top\]
Symbols: \(\det(A)\) = determinant; \(A^{-\top}\) = inverse transpose.
Intuition: Appears in the log-likelihood of Gaussian distributions where the normalizing constant contains log det(Σ). For symmetric A: simplifies to A^{-1}.

Question 48 AI Math (500 Questions)

What is the difference between transposed convolution and upsampling?

\[\mathbb{E}_\theta[\hat{\theta}(X)] = \theta \text{ for all } \theta\]
Symbols: \(\hat{\theta}\) = estimator function of data X; equality holds for all possible true values.
Intuition: An unbiased estimator's sampling distribution is centered at the true parameter. The sample mean is unbiased for μ; MLE variance estimate is biased (use Bessel's correction). \[W \sim \mathcal{N}\!\left(0,\, \frac{2}{n_{\text{in}}+n_{\text{out}}}\right)\]
Symbols: \(n_{\text{in}}, n_{\text{out}}\) = number of input and output units; variance chosen to preserve signal variance.
Intuition: Designed for tanh/sigmoid activations. Sets variance to keep gradient magnitudes roughly constant across layers — prevents vanishing/exploding. Derived by requiring Var(output) = Var(input) in forward pass. \[\text{TransposedConv: learnable upsampling; Upsample: nearest/bilinear (fixed)}\]
Symbols: TransposedConv = fractionally-strided convolution; increases spatial resolution.
Intuition: Used in decoders and generators (GANs, autoencoders) to increase spatial resolution. Transposed conv learns the upsampling; bilinear interpolation is a fixed mathematical operation. Transposed conv can produce 'checkerboard' artifacts. \[\text{softmax}(\mathbf{z}/T)_i = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}}\]
Symbols: \(T\) = temperature; \(T<1\) sharpens; \(T>1\) flattens distribution.
Intuition: Low T → near one-hot (model more confident/decisive). High T → near uniform (more exploratory/uncertain). Used in knowledge distillation, sampling from LLMs, and reinforcement learning.

Question 49 AI Math (500 Questions)

What is learning rate warmup?

\[\eta_t = \eta_{\max} \cdot \frac{t}{T_{\mathrm{warm}}} \text{ for } t \leq T_{\mathrm{warm}}\]
Symbols: \(T_{\mathrm{warm}}\) = warmup steps; linearly increases LR from 0 to η_max.
Intuition: Avoids large steps at the start when parameters are randomly initialized (moments in Adam are unreliable early on). Warmup + cosine decay is the standard transformer training schedule. \[\mathcal{L}_{\mathrm{reg}} = \mathcal{L} + \lambda\sum_j |w_j|\]
Symbols: \(\lambda\) = regularization strength; L1 subgradient: \(\mathrm{sign}(w_j)\).
Intuition: Penalizes the sum of absolute weight values. Produces sparse models — many weights driven exactly to zero (unlike L2 which only shrinks them). Equivalent to Laplace prior. Useful for feature selection. \[p(x|\lambda) = \lambda e^{-\lambda x}, \quad x \geq 0, \quad \mathbb{E}[X] = 1/\lambda\]
Symbols: \(\lambda > 0\) = rate parameter; special case of Gamma with shape k=1.
Intuition: Models time until first event in a Poisson process. Memoryless property. Used in survival analysis, network queuing, and modeling inter-arrival times. \[f \text{ convex} \iff \nabla^2 f(\mathbf{x}) \succeq 0 \text{ for all } \mathbf{x}\]
Symbols: \(\nabla^2 f\) = Hessian; \(\succeq 0\) = positive semi-definite.
Intuition: A function is convex iff its Hessian is PSD everywhere — non-negative curvature in all directions. Strictly convex has PD Hessian and a unique global minimum.

Question 50 AI Math (500 Questions)

What does it mean for vectors to be linearly independent?

\[D_{\mathrm{KL}}(q\|p) = -\frac{1}{2}\sum_{j=1}^d \left(1 + \log\sigma_j^2 - \mu_j^2 - \sigma_j^2\right)\]
Symbols: \(d\) = latent dimension; \(\mu_j, \sigma_j^2\) = variational parameters per dimension.
Intuition: Closed-form regularization term in VAE training. Pushes posterior toward the standard Gaussian prior — prevents the encoder from ignoring the prior. \[\mathbf{z} = W^\top (\mathbf{x} - \boldsymbol{\mu}), \quad W = [\mathbf{u}_1, \ldots, \mathbf{u}_k]\]
Symbols: \(\mathbf{x}\) = data point; \(\boldsymbol{\mu}\) = mean; \(\mathbf{u}_i\) = i-th principal component (eigenvector of covariance matrix); \(k\) = reduced dimension.
Intuition: Projects data onto the k directions of maximum variance. Finds a low-dimensional subspace that retains as much information as possible. \[c_1\mathbf{v}_1 + c_2\mathbf{v}_2 + \cdots + c_k\mathbf{v}_k = \mathbf{0} \implies c_1 = c_2 = \cdots = c_k = 0\]
Symbols: \(c_i \in \mathbb{R}\) = scalar coefficients; \(\mathbf{0}\) = zero vector.
Intuition: No vector in the set can be written as a combination of the others. Linearly independent vectors span the largest possible subspace. \[\mathrm{rank}(A) = \#\{\sigma_i > 0\}\]
Symbols: \(\sigma_i\) = singular values.
Intuition: The rank equals the number of non-zero singular values. Singular values near zero indicate near-redundancy in the data — used for dimensionality assessment.