Subspace concepts reveal the structure of linear transformations. The span of vectors \(\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}\) consists of all linear combinations \(\sum_i c_i \mathbf{v}_i\). Vectors are linearly independent iff the only combination giving zero is the trivial one (all coefficients zero). A basis is a linearly independent set that spans the space—any basis of \(\mathbb{R}^n\) has exactly \(n\) vectors. The rank \(\mathrm{rank}(A) = \dim(\mathrm{col}(A)) = \dim(\mathrm{row}(A))\) counts linearly independent columns (or rows), equals the number of nonzero singular values, and is full iff \(\mathrm{rank} = \min(m,n)\). The column space \(\mathrm{col}(A) = \{A\mathbf{x}\}\) contains all outputs of A, and \(Ax = b\) has a solution iff \(b\) lies in \(\mathrm{col}(A)\). The null space (kernel) \(\ker(A) = \{\mathbf{x}: A\mathbf{x} = \mathbf{0}\}\) contains all inputs A maps to zero, and the rank-nullity theorem states \(\mathrm{rank}(A) + \mathrm{nullity}(A) = n\)—the input dimensions split into mapped and collapsed parts. Rank-deficient matrices have \(\mathrm{rank}(A) < \min(m,n)\), implying singular columns and \(\det(A) = 0\).
Eigenvalues and eigenvectors capture how matrices act in special directions. The eigenvalue equation \(A\mathbf{v} = \lambda\mathbf{v}\) with \(\mathbf{v} \neq \mathbf{0}\) finds directions where A only stretches (by \(\lambda\)) without rotating—positive \(\lambda\) keeps direction, negative flips, zero collapses. To find eigenvalues, solve the characteristic polynomial \(\det(A - \lambda I) = 0\), giving exactly \(n\) eigenvalues (counting multiplicity). Distinct eigenvalues always produce linearly independent eigenvectors. The eigenspace \(E_\lambda = \ker(A - \lambda I)\) collects all eigenvectors for \(\lambda\) plus zero; its dimension is the geometric multiplicity. Symmetric matrices have real eigenvalues with orthogonal eigenvectors (spectral theorem). Eigendecomposition factors \(A = Q\Lambda Q^{-1}\) where Q holds eigenvectors and \(\Lambda\) holds eigenvalues; for symmetric matrices \(A = Q\Lambda Q^\top\) with orthogonal Q. This enables easy computation of matrix powers \(A^k = Q\Lambda^k Q^{-1}\) and matrix exponentials \(e^A = Q e^\Lambda Q^{-1}\). The trace equals the sum and determinant equals the product of eigenvalues—quick sanity checks on computations.
Matrix decompositions generalize eigendecomposition. The Singular Value Decomposition (SVD) factors any matrix \(A = U\Sigma V^\top\) with orthogonal U, V and diagonal \(\Sigma\) of non-negative singular values \(\sigma_i \geq 0\). These relate to eigenvalues via \(\sigma_i(A) = \sqrt{\lambda_i(A^\top A)}\), measuring how much A stretches space in each direction. The economy SVD keeps only nonzero singular values. By the Eckart-Young theorem, the best rank-k approximation is \(A_k = \sum_{i=1}^k \sigma_i \mathbf{u}_i \mathbf{v}_i^\top\) with reconstruction error \(\|A - A_k\|_F = \sqrt{\sum_{i>k} \sigma_i^2}\)—this drives dimensionality reduction, recommendation systems via latent factors \(\hat{R} \approx U_k \Sigma_k V_k^\top\), and Principal Component Analysis (PCA), which projects onto the k directions of maximum variance. A symmetric positive definite matrix can be Cholesky-factored \(A = LL^\top\) with lower-triangular L, useful for efficient linear system solving. The QR decomposition \(A = QR\) with orthonormal columns in Q and upper-triangular R comes from the Gram-Schmidt process. Positive definite matrices \(A \succ 0\) have all \(\lambda_i > 0\) (bowl-shaped curvature, unique minimum); positive semi-definite (PSD) matrices \(A \succeq 0\) have \(\lambda_i \geq 0\) (covariance matrices are always PSD, with precision matrix \(\Lambda = \Sigma^{-1}\) encoding conditional independence). The Moore-Penrose pseudoinverse \(A^+ = V\Sigma^+ U^\top\) generalizes the inverse to non-square or singular matrices, providing minimum-norm least-squares solutions. Additional matrix norms include the spectral norm \(\|A\|_2 = \sigma_{\max}(A)\) (maximum stretching, used in GAN spectral normalization) and nuclear norm \(\|A\|_* = \sum_i \sigma_i\) (convex relaxation of rank, used in matrix completion).