Linear Algebra Essentials

It's well suited for students taking their first linear algebra course, learners reviewing for exams, or anyone returning to the subject who wants to refresh their understanding of key terminology. Because linear algebra is cumulative, getting these definitions clear early on tends to make later material much easier to follow.

To get the most out of this deck, try studying in short, focused sessions rather than long cramming blocks, since spaced repetition helps these definitions move into long-term memory. It can also help to group related cards together as you review, so you start seeing how ideas like span, basis, and linear independence connect to one another. Revisiting any cards you find tricky across multiple days will strengthen your grasp of the underlying language of the subject.

A scalar is a single real or complex number, the simplest object in linear algebra. Building from it, a vector is an ordered list of scalars, which can be visualized geometrically as a directed magnitude in space. A matrix generalizes this further as a rectangular array of numbers arranged in rows and columns, while a tensor is the most general object: a multi-dimensional array whose order ranges from scalars (0-D) to vectors (1-D), matrices (2-D), and higher.

A vector space is a set V equipped with two operations, vector addition and scalar multiplication, satisfying eight axioms: closure under both operations, commutativity and associativity of addition, existence of an additive identity (the zero vector) and additive inverses, a multiplicative identity, and distributivity of scalar multiplication over both vector addition and scalar addition. The difference u − v is then defined as u + (−v), ensuring that subtraction, like addition and scaling, stays inside the space.

Important structural concepts arise from how vectors can be combined. The span of a set of vectors is the set of all their linear combinations \(c_1 v_1 + c_2 v_2 + \cdots + c_n v_n\), and it always forms a subspace. A subspace more generally is any non-empty subset closed under addition and scalar multiplication that contains the zero vector. A set of vectors is linearly independent when no vector in it can be written as a linear combination of the others — equivalently, the only combination that gives the zero vector is the trivial one with all coefficients zero; otherwise the set is linearly dependent.

A basis is a linearly independent spanning set, and the dimension of a vector space is the number of vectors in any basis (every basis has the same size). In \(\mathbb{R}^n\) the standard basis consists of the vectors \(e_1, \ldots, e_n\) with a single 1 in one coordinate and zeros elsewhere. Every vector space has a basis, a statement equivalent to the axiom of choice.

A matrix is a rectangular array of numbers. Its transpose \(A^T\) is obtained by swapping rows and columns, so \((A^T)_{ij} = A_{ji}\). The inverse \(A^{-1}\) of a square matrix is the unique matrix satisfying \(AA^{-1} = A^{-1}A = I\), where the identity matrix \(I\) has 1s on the diagonal and 0s elsewhere. Inversion is possible exactly when \(\det(A) \neq 0\).

Many special matrices arise repeatedly. A diagonal matrix has nonzero entries only on the main diagonal, while an upper (or lower) triangular matrix has zero entries below (or above) the diagonal. A symmetric matrix satisfies \(A = A^T\), and a skew-symmetric matrix satisfies \(A^T = -A\), forcing zero diagonal entries. An orthogonal matrix \(Q\) satisfies \(Q^T Q = Q Q^T = I\), meaning its columns form an orthonormal set. The complex analogue is the unitary matrix, where \(U^*U = UU^* = I\) with \(U^*\) denoting the conjugate transpose. A matrix is normal when \(AA^* = A^*A\); normal matrices are precisely those that can be unitarily diagonalized. A Hermitian matrix is the complex analogue of a symmetric one, equal to its own conjugate transpose. A symmetric matrix is positive definite if \(x^T A x > 0\) for every nonzero real vector \(x\), and positive semidefinite when the inequality is non-strict; Sylvester's criterion gives an alternative test by requiring all leading principal minors to be positive.

A linear transformation is a function \(T: V \to W\) between vector spaces that respects the structure: \(T(u+v) = T(u) + T(v)\) and \(T(cv) = c T(v)\). Once bases are chosen for \(V\) and \(W\), \(T\) is represented by a matrix \(A\) with \(T(x) = Ax\); the columns of \(A\) are the images of the basis vectors of \(V\). The product of two matrices corresponds to the composition of the corresponding linear maps. Two matrices \(A\) and \(B\) are similar when \(A = P^{-1} B P\) for some invertible \(P\); similar matrices represent the same linear map in different bases, and they share the same characteristic polynomial, trace, determinant, rank, and eigenvalues. The change-of-basis matrix from basis \(B\) to basis \(C\) has columns equal to the \(C\)-coordinates of the \(B\) basis vectors, converting \(B\)-coordinates into \(C\)-coordinates.

The determinant is a scalar that captures key information about a square matrix. For the 2×2 matrix \(\begin{pmatrix} a & b \\ c & d \end{pmatrix}\) it is simply \(ad - bc\). Geometrically, the determinant measures how the associated linear map scales area (in two dimensions) or volume (in higher dimensions); a zero determinant signals that the matrix is singular — its columns are linearly dependent and the map collapses dimension.

Determinants obey several useful identities. They are multiplicative, \(\det(AB) = \det(A)\det(B)\), and invariant under transposition, \(\det(A^T) = \det(A)\). The determinant of an inverse is \(\det(A^{-1}) = 1/\det(A)\). Swapping two rows flips the sign of the determinant, and for a triangular matrix the determinant is simply the product of its diagonal entries. These properties make \(\det(A) = 0\) a clean test for non-invertibility and let us derive many determinant formulas from a few basic cases.

For larger matrices, determinants are computed by cofactor expansion: \(\det(A) = \sum_j (-1)^{i+j} a_{ij} M_{ij}\) along any fixed row \(i\), where \(M_{ij}\) is the \((i,j)\) minor, namely the determinant of the submatrix obtained by deleting row \(i\) and column \(j\). Cramer's rule then expresses each component of the solution to \(Ax = b\) as a ratio \(x_i = \det(A_i)/\det(A)\), where \(A_i\) is the matrix obtained by replacing the \(i\)-th column of \(A\) with \(b\). The Cayley–Binet formula generalizes a piece of this: for \(A\) of size \(m \times n\) and \(B\) of size \(n \times m\), \(\det(AB)\) equals the sum of the determinants of all \(n \times n\) submatrices formed by choosing \(n\) columns of \(A\) together with the corresponding rows of \(B\). A related differentiation rule concerns time-dependent matrices: \(\frac{d}{dt} \det(A(t)) = \det(A) \cdot \mathrm{tr}(A^{-1} A')\), tying the rate of change of the determinant to the trace.

An eigenvalue of a square matrix \(A\) is a scalar \(\lambda\) for which \(Av = \lambda v\) for some nonzero vector \(v\); the corresponding vector \(v\) is the eigenvector. Eigenvalues encode how \(A\) acts along preferred directions and are precisely the roots of the characteristic polynomial \(p(\lambda) = \det(A - \lambda I)\). For a 2×2 matrix \(\begin{pmatrix} a & b \\ c & d \end{pmatrix}\) this polynomial becomes \(\lambda^2 - (a+d)\lambda + (ad-bc) = 0\); the sum \(a+d\) is therefore the trace, while the constant \(ad-bc\) is the determinant.

A matrix is diagonalizable when it can be written \(A = P D P^{-1}\) with \(D\) diagonal; the columns of \(P\) are the eigenvectors of \(A\), and \(D\) contains the corresponding eigenvalues. The criterion is clean: diagonalizability holds exactly when \(A\) has \(n\) linearly independent eigenvectors, which happens when the geometric multiplicity of every eigenvalue — the dimension of its eigenspace, the null space of \(A - \lambda I\) — equals its algebraic multiplicity, namely the multiplicity of \(\lambda\) as a root of the characteristic polynomial. The trace equals the sum of eigenvalues, and the Cayley–Hamilton theorem guarantees that every square matrix satisfies its own characteristic polynomial: \(p(A) = 0\).

Several canonical forms sharpen this picture. The spectral theorem says that every real symmetric matrix can be orthogonally diagonalized, \(A = Q D Q^T\) with \(Q\) orthogonal and \(D\) real diagonal. The Schur decomposition relaxes the symmetry requirement to write any square matrix as \(A = Q T Q^*\) with \(Q\) unitary and \(T\) upper triangular; the diagonal entries of \(T\) are the eigenvalues. The polar decomposition goes further by writing a square invertible matrix as \(A = UP\) with \(U\) unitary and \(P\) positive definite. When a matrix is not diagonalizable, the Jordan canonical form provides the closest analogue: every square matrix is similar to an upper-triangular matrix whose diagonal entries are the eigenvalues and whose superdiagonal contains 1s grouped into Jordan blocks. The minimal polynomial is the monic polynomial of least degree annihilating \(A\); for a diagonalizable matrix it is the product \(\prod_i (x - \lambda_i)\) of distinct factors over its eigenvalues. The companion matrix of a monic polynomial \(p(x) = x^n + c_{n-1} x^{n-1} + \cdots + c_0\) is an \(n \times n\) matrix whose characteristic polynomial is \(p\), giving a concrete matrix realization of any monic polynomial.

The inner (dot) product of two real vectors \(u\) and \(v\) is \(u \cdot v = \sum_i u_i v_i = u^T v\); it equals zero precisely when \(u\) and \(v\) are orthogonal. From it comes the Euclidean norm \(\|x\| = \sqrt{x \cdot x} = \sqrt{\sum_i x_i^2}\). The Cauchy–Schwarz inequality \(|u \cdot v| \leq \|u\| \|v\|\) follows naturally, with equality if and only if \(u\) and \(v\) are linearly dependent. Hölder's inequality extends this to conjugate exponents: for \(1/p + 1/q = 1\), one has \(\sum_i |u_i v_i| \leq \|u\|_p \, \|v\|_q\).

A vector norm in general is a function on a space that is nonnegative, vanishes only at the zero vector, scales by scalars \(\|c x\| = |c| \|x\|\), and satisfies the triangle inequality \(\|u + v\| \leq \|u\| + \|v\|\). For matrices, the Frobenius norm is \(\|A\|_F = \sqrt{\sum_{i,j} A_{ij}^2} = \sqrt{\sum \sigma_i^2}\) where \(\sigma_i\) are the singular values, while the L2 or spectral norm \(\|A\|_2\) is the largest singular value — the maximum factor by which \(A\) stretches any unit vector.

The condition number \(\kappa(A) = \|A\| \cdot \|A^{-1}\|\), equal in the 2-norm to the ratio of the largest to smallest singular value, captures how badly a linear system amplifies perturbations. A large condition number means that small changes in \(b\) can produce large changes in \(x = A^{-1} b\), making numerical solutions unreliable. The spectral radius \(\rho(A)\) is the largest absolute eigenvalue, and Gershgorin's circle theorem locates every eigenvalue in at least one disc centered at a diagonal entry \(a_{ii}\) with radius equal to the sum of absolute off-diagonal entries in row \(i\).

When given a basis that is not orthogonal, one can build one that is. The Gram–Schmidt process orthogonalizes a basis by successively subtracting projections onto previously constructed vectors and normalizing; it produces a QR decomposition \(A = QR\) where \(Q\) has orthonormal columns and \(R\) is upper triangular. Vectors that are mutually orthogonal and each of unit length are called orthonormal, and such collections form natural coordinate systems in which calculations mimic those in the standard basis.

Factorizations express complicated matrices as products of simpler ones, exposing structure and enabling efficient computation. The LU decomposition writes a square matrix as \(A = LU\) with \(L\) lower triangular and \(U\) upper triangular, useful for solving linear systems by sequential substitution. It exists without row pivoting precisely when every leading principal minor of \(A\) is nonzero. The Cholesky decomposition strengthens this for symmetric positive-definite matrices, writing \(A = LL^T\) with \(L\) lower triangular having positive diagonal entries.

The singular value decomposition (SVD) applies to every matrix, regardless of shape. Any \(m \times n\) matrix \(A\) can be written \(A = U \Sigma V^T\) with \(U\) and \(V\) orthogonal and \(\Sigma\) a rectangular diagonal matrix whose entries, the singular values, are the square roots of the eigenvalues of \(A^T A\), arranged in non-increasing order. The number of nonzero singular values equals the rank of \(A\). Associated with the SVD is the Moore–Penrose pseudoinverse \(A^+\), defined by the four Penrose conditions; for a full-rank \(A\) it equals \((A^T A)^{-1} A^T\).

These decompositions provide the foundation for least squares. The least-squares solution to \(Ax = b\) is the vector \(x\) that minimizes \(\|Ax - b\|^2\); it satisfies the normal equations \(A^T A x = A^T b\), whose unique solution (when \(A^T A\) is invertible) is \(x = (A^T A)^{-1} A^T b = A^+ b\). Projections also play a key role: a projection matrix \(P\) satisfies \(P^2 = P\), and its image is precisely the subspace onto which it projects. An orthogonal projection matrix additionally satisfies \(P = P^T\), projecting onto its column space along the orthogonal complement.

For systems of linear equations, row operations — swapping two rows, multiplying a row by a nonzero scalar, or adding a multiple of one row to another — transform a matrix without changing the solution set of \(Ax = b\). Repeated application produces row echelon form, where each leading entry of a row lies to the right of that of the row above and zero rows sit at the bottom, and the further-normalized reduced row echelon form (RREF), in which each leading 1 is the only nonzero entry in its column. The Rank–Nullity theorem ties everything together: for an \(m \times n\) matrix \(A\), \(\mathrm{rank}(A) + \mathrm{nullity}(A) = n\). Rank is the dimension of the column (or row) space, and nullity is the dimension of the null space, the set of solutions to \(Ax = 0\).

Applications appear throughout the sciences. In probability, a stochastic matrix has nonnegative entries whose rows (or columns) sum to 1; the row-stochastic version serves as the transition matrix of a Markov chain, with entry \(P_{ij}\) giving the probability of moving from state \(i\) to state \(j\). The steady state \(\pi\) is the probability vector satisfying \(\pi P = \pi\) and \(\sum_i \pi_i = 1\) — an eigenvector of \(P^T\) with eigenvalue 1. The Perron–Frobenius theorem guarantees that a positive square matrix has a unique largest real eigenvalue, the Perron root, with a positive eigenvector, and that the spectral radius is a simple eigenvalue. In statistics, the covariance matrix \(\Sigma = E[(X - \mu)(X - \mu)^T]\) of a random vector is symmetric and positive semidefinite, and PCA uses the eigendecomposition of \(\Sigma\): the principal components are eigenvectors of \(\Sigma\), with eigenvalues giving the variance explained along each direction. Symmetric matrices are particularly important in applications because they have real eigenvalues and orthogonal eigenvectors, they arise naturally in quadratic forms and mechanical systems, and they are numerically stable to diagonalize.

Quadratic forms \(Q(x) = x^T A x\), defined by symmetric matrices, classify definiteness through their sign on nonzero vectors. The signature of a symmetric matrix records the count of positive, negative, and zero eigenvalues, while the inertia is the triple \((n_+, n_-, n_0)\) of these counts. Sylvester's law of inertia states that two real symmetric matrices are congruent — that is, \(B = S^T A S\) for some invertible \(S\) — precisely when they share the same inertia. Congruence preserves definiteness but not eigenvalues.

A smooth counterpart to all of this is the matrix exponential \(\exp(A) = \sum_k A^k / k!\), which converges for every square matrix \(A\) and solves the differential equation \(dX/dt = A X\) with \(X(0) = I\). The Jacobian matrix of a vector-valued function \(f: \mathbb{R}^n \to \mathbb{R}^m\) has entries \(J_{ij} = \partial f_i / \partial x_j\), and the determinant of the Jacobian represents the local volume scaling factor of the transformation near a point, neatly linking the geometry of multivariable calculus with the algebra of determinants.