Derivatives measure instantaneous rates of change, forming the foundation of optimization in machine learning. The derivative \(f'(x) = \lim_{h \to 0}(f(x+h)-f(x))/h\) is the slope of the tangent line, equivalently written as \(df/dx\) or \(\dot{f}\) in different notations. Basic rules include the constant rule (\(d/dx[c] = 0\)), power rule (\(d/dx[x^n] = nx^{n-1}\)), sum rule (\(d/dx[f+g] = f'+g'\), reflecting linearity), product rule (\(d/dx[fg] = f'g + fg'\)), and quotient rule (\(d/dx[f/g] = (f'g - fg')/g^2\)). The chain rule for composite functions \(d/dx[f(g(x))] = f'(g(x)) g'(x)\) extends to multiple compositions \(d/dx[f_1(f_2(\cdots f_n(x)))] = f_1' \cdot f_2' \cdots f_n'\)—backpropagation in deep networks is precisely this chain rule applied repeatedly. Key derivatives include \(d/dx[e^x] = e^x\) (the exponential is its own derivative), \(d/dx[\ln x] = 1/x\), \(d/dx[\sigma(x)] = \sigma(x)(1-\sigma(x))\) for sigmoid, and \(d/dx[\tanh(x)] = 1 - \tanh^2(x)\) for tanh. The log-derivative trick \(\partial \log p(x|\theta)/\partial\theta = (1/p)\partial p/\partial\theta\) underlies MLE and the REINFORCE estimator \(\nabla_\theta \mathbb{E}_{p_\theta}[f(x)] = \mathbb{E}_{p_\theta}[f(x)\nabla_\theta \log p_\theta(x)]\).
Multivariable calculus extends these ideas. Partial derivatives \(\partial f/\partial x_i\) measure change in one direction while holding others constant. The gradient \(\nabla f(\mathbf{x}) = [\partial f/\partial x_1, \ldots, \partial f/\partial x_n]^\top\) is the vector of partial derivatives, pointing in the direction of steepest ascent—so \(-\nabla f\) points downhill (the basis of gradient descent). Key gradient identities include \(\nabla(\mathbf{w}^\top\mathbf{x}) = \mathbf{w}\) for linear functions and \(\nabla(\mathbf{x}^\top A \mathbf{x}) = (A + A^\top)\mathbf{x}\), which simplifies to \(2A\mathbf{x}\) for symmetric A (analogous to the scalar derivative of \(ax^2\)). The gradient of the squared L2 norm is \(\nabla\|\mathbf{x}\|_2^2 = 2\mathbf{x}\). The directional derivative \(D_\mathbf{u} f = \nabla f \cdot \mathbf{u}\) gives the rate of change in any unit direction, maximized when aligned with \(\nabla f\). The Jacobian matrix \(J_{ij} = \partial f_i/\partial x_j\) generalizes the gradient to vector-valued functions, representing the best linear approximation near a point: \(\mathbf{f}(\mathbf{x}+\delta) \approx \mathbf{f}(\mathbf{x}) + J\delta\). The Jacobian determinant's absolute value gives the volume scaling factor of the transformation—central in change-of-variable formulas for probability densities and normalizing flows \(p_x(\mathbf{x}) = p_z(\mathbf{f}(\mathbf{x}))|\det J|\). The multivariate chain rule expresses composition via Jacobian multiplication: \(\partial \mathbf{f}/\partial \mathbf{x} = (\partial \mathbf{f}/\partial \mathbf{g})(\partial \mathbf{g}/\partial \mathbf{x})\)—exactly backpropagation.
Second-order information describes curvature. The Hessian \(H_{ij} = \partial^2 f/\partial x_i \partial x_j\) is the symmetric matrix of second partial derivatives. At a critical point \(\nabla f(\mathbf{x}^*) = \mathbf{0}\): positive definite Hessian (\(H \succ 0\), all eigenvalues positive) implies a local minimum (bowl shape), negative definite implies local maximum, and indefinite Hessian (mixed signs) implies a saddle point—common in high-dimensional loss landscapes. The Hessian condition number \(\kappa(H) = \lambda_{\max}/\lambda_{\min}\) characterizes how much curvature differs across directions; high values create narrow valleys where gradient descent oscillates. Taylor expansion provides local approximations: \(f(\mathbf{x}+\boldsymbol{\delta}) \approx f(\mathbf{x}) + \nabla f^\top \boldsymbol{\delta}\) (first-order, used to show \(\mathcal{L}(\theta-\eta\nabla\mathcal{L}) \approx \mathcal{L}(\theta) - \eta\|\nabla\mathcal{L}\|^2\)) and the second-order form \(f(\mathbf{x}+\boldsymbol{\delta}) \approx f(\mathbf{x}) + \nabla f^\top \boldsymbol{\delta} + \frac{1}{2}\boldsymbol{\delta}^\top H \boldsymbol{\delta}\) (basis for Newton's method \(\mathbf{x}_{t+1} = \mathbf{x}_t - H^{-1}\nabla f(\mathbf{x}_t)\)). A function is convex iff its Hessian is PSD everywhere—the second-order condition for convexity. L-smooth functions satisfy \(\|\nabla f(\theta_1) - \nabla f(\theta_2)\| \leq L\|\theta_1-\theta_2\|\) (bounded gradient change). Lipschitz continuous functions \(|f(x) - f(y)| \leq L|x-y|\) have bounded slope. Subgradients generalize gradients to non-smooth functions like ReLU and \(|x|\): at the kink of \(|x|\) at zero, any value in \([-1,1]\) is a valid subgradient—this is what allows L1 regularization to produce sparsity. The mean value theorem states some point achieves the average slope; L'Hôpital's rule \(\lim f/g = \lim f'/g'\) handles indeterminate forms \(0/0\) or \(\infty/\infty\).