Vector space structure forms the foundation of all machine learning mathematics. A vector \(\mathbf{x} = [x_1, \ldots, x_n]^\top \in \mathbb{R}^n\) is an ordered list of \(n\) numbers that represents a data point, feature embedding, or parameter set as a point in \(n\)-dimensional space. Vectors support element-wise addition \([a_1+b_1, \ldots, a_n+b_n]^\top\) and scalar multiplication \(c\mathbf{x} = [cx_1, \ldots, cx_n]^\top\), which geometrically stretches, shrinks, or flips the vector. Measuring magnitude uses norms: the L2 norm \(\|\mathbf{x}\|_2 = \sqrt{\sum_i x_i^2}\) is the Euclidean distance from the origin, the L1 norm \(\|\mathbf{x}\|_1 = \sum_i |x_i|\) (Manhattan distance) encourages sparsity in regularization, and the general Lp norm \(\|\mathbf{x}\|_p = (\sum_i |x_i|^p)^{1/p}\) interpolates between them, becoming the max norm as \(p \to \infty\). The dot product \(\mathbf{a} \cdot \mathbf{b} = \sum_i a_i b_i = \|\mathbf{a}\|_2\|\mathbf{b}\|_2 \cos\theta\) measures alignment between vectors—orthogonal iff zero, with cosine similarity \(\cos\theta = (\mathbf{a}\cdot\mathbf{b})/(\|\mathbf{a}\|_2\|\mathbf{b}\|_2)\) giving a normalized \([-1,1]\) score. The Cauchy-Schwarz inequality bounds the dot product by the product of norms: \(|\mathbf{a} \cdot \mathbf{b}| \leq \|\mathbf{a}\|_2 \|\mathbf{b}\|_2\), with equality iff the vectors are parallel.
Matrix operations extend vectors into rectangular arrays. A matrix \(A \in \mathbb{R}^{m \times n}\) has \(m\) rows and \(n\) columns, with \(A_{ij}\) indexing the entry at row \(i\) and column \(j\). Addition is element-wise \((A+B)_{ij} = A_{ij}+B_{ij}\) for same-shape matrices, and scalar multiplication scales every entry. Matrix multiplication \(C = AB\) with \(A \in \mathbb{R}^{m \times p}\) and \(B \in \mathbb{R}^{p \times n}\) produces \(C \in \mathbb{R}^{m \times n}\) via \(C_{ij} = \sum_k A_{ik}B_{kj}\)—each entry is a dot product of a row of A with a column of B, representing composition of linear transformations. This multiplication is non-commutative (\(AB \neq BA\) in general) but associative (\((AB)C = A(BC)\)). The matrix-vector product \(A\mathbf{x} = \sum_j x_j \mathbf{a}_j\) is a linear combination of A's columns weighted by \(\mathbf{x}\)'s components, which is the foundation of linear layers in neural networks.
Special matrices and transformations include the transpose \((A^\top)_{ij} = A_{ji}\) that flips rows and columns with the key identity \((AB)^\top = B^\top A^\top\) (order reverses). A symmetric matrix satisfies \(A = A^\top\) and always has real eigenvalues (covariance matrices and Hessians are symmetric). The identity matrix \(I\) has 1s on the diagonal and 0s elsewhere, serving as the multiplicative identity (\(AI = IA = A\)), while the zero matrix has all zero entries and acts as the additive identity. The Frobenius norm \(\|A\|_F = \sqrt{\sum_{ij} A_{ij}^2}\) applies the L2 norm to a flattened matrix, equivalently \(\sqrt{\mathrm{tr}(A^\top A)}\).
Scalar matrix functions complete the algebraic toolkit. The trace \(\mathrm{tr}(A) = \sum_i A_{ii}\) sums diagonal entries, equals the sum of eigenvalues, and obeys the cyclic property \(\mathrm{tr}(ABC) = \mathrm{tr}(BCA) = \mathrm{tr}(CAB)\) (non-cyclic permutations are not equal in general). The determinant for a 2×2 matrix \(\det\begin{pmatrix}a & b \\ c & d\end{pmatrix} = ad - bc\) is the signed area of the parallelogram formed by column vectors—zero meaning singular. Its absolute value is the volume scaling factor of the linear map, the product rule \(\det(AB) = \det(A)\det(B)\) makes volume scaling multiplicative, and the determinant equals the product of eigenvalues. The inverse \(A^{-1}\) satisfying \(AA^{-1} = A^{-1}A = I\) exists iff \(\det(A) \neq 0\), with \((AB)^{-1} = B^{-1}A^{-1}\); orthogonal matrices (those with \(Q^\top Q = I\)) have inverse equal to their transpose. The Hadamard product \((A \odot B)_{ij} = A_{ij} B_{ij}\) multiplies corresponding elements and appears in attention masks, dropout masks, and LSTM gate computations.