Multivariable calculus extends differentiation to functions of many variables. The partial derivative \(\partial f/\partial x_i\) measures how \(f\) changes when only \(x_i\) moves; the gradient \(\nabla f\) collects all partial derivatives into a vector that points in the direction of steepest ascent, with magnitude equal to the rate of increase. To minimize a function, you move in the opposite direction, which is exactly why gradient descent updates parameters as \(\theta_{t+1} = \theta_t - \eta \nabla L(\theta_t)\). The Hessian matrix collects all second partial derivatives and encodes the local curvature: at a critical point, a positive-definite Hessian indicates a local minimum, a negative-definite Hessian indicates a local maximum, and mixed eigenvalues signal a saddle point.
The chain rule generalizes to several variables through the Jacobian matrix, whose entries are the first-order partial derivatives of a vector-valued function. Composing functions multiplies their Jacobians. In backpropagation, this chain is unrolled layer by layer, multiplying local derivatives from the loss all the way back to each weight. Automatic differentiation implements this automatically by tracking operations in a computational graph, computing exact derivatives up to machine precision in either forward or reverse mode, with reverse mode being the natural choice for backpropagation since it handles many parameters efficiently.
Linear algebra supplies the language of vectors, matrices, and the transformations between them. A matrix encodes a linear map, and matrix multiplication composes such maps. Special matrices include the identity, diagonal, triangular, symmetric, orthogonal, and positive-definite matrices, each with useful properties. The determinant measures how a transformation scales volumes, and a matrix is invertible precisely when its determinant is nonzero. Eigenvalues and eigenvectors characterize how a matrix acts along its principal directions: \(A\mathbf{v} = \lambda \mathbf{v}\). Diagonalization writes a matrix as \(A = PDP^{-1}\), and the spectral theorem guarantees that every symmetric matrix can be orthogonally diagonalized with real eigenvalues.
The singular value decomposition (SVD) factors any matrix \(A = U\Sigma V^\top\) and generalizes eigendecomposition to non-square matrices. Truncating the SVD yields the best low-rank approximation in the Frobenius norm, which underlies principal component analysis (PCA), recommender systems, and matrix factorization. Optimization theory builds on convexity: a convex function has the property that every local minimum is also a global minimum, so gradient descent is guaranteed to find the best solution. For non-convex problems common in deep learning, variants like SGD, momentum, Adam, and learning rate schedules (warmup, cosine annealing) help navigate complex loss landscapes. Regularization adds a penalty term to the loss, with L1 promoting sparsity, L2 promoting small weights, and dropout providing implicit regularization through stochastic network perturbations.