Self-information (surprisal) \(I(x) = -\log_2 p(x)\) measures surprise: rare events carry more information. Doubling rarity adds exactly 1 bit, since log converts products to sums, making information additive for independent events: \(I(x_1 \cap x_2) = I(x_1) + I(x_2)\). Units vary by log base: \(-\log_2\) gives bits (shannons), \(-\ln\) gives nats, \(-\log_{10}\) gives hartleys, with 1 nat ≈ 1.44 bits. Shannon entropy \(H(X) = -\sum_x p(x)\log_2 p(x)\) is the expected self-information—average surprise. It is maximized at \(\log_2 K\) bits when all K outcomes are equally likely (maximum uncertainty) and is zero when the outcome is certain. Binary entropy \(H(p) = -p\log_2 p - (1-p)\log_2(1-p)\) reaches 1 bit at \(p=0.5\) and zero at \(p \in \{0,1\}\)—used in decision tree splitting criteria. Differential entropy \(h(X) = -\int p(x)\ln p(x)dx\) for continuous variables can be negative and is reparameterization-dependent; for a Gaussian: \(h = \frac{1}{2}\ln(2\pi e\sigma^2)\) nats. The maximum entropy principle chooses the least-committed distribution subject to moment constraints: with mean and variance fixed, the result is the Gaussian; with no constraints, the uniform distribution.
Joint and conditional entropy decompose uncertainty. Joint entropy \(H(X,Y) = -\sum_{x,y} p(x,y)\log p(x,y)\) decomposes via the chain rule \(H(X_1,\ldots,X_n) = \sum_i H(X_i|X_1,\ldots,X_{i-1})\), with \(H(X,Y) = H(Y) + H(X|Y)\) for two variables. Conditional entropy \(H(X|Y) = -\sum_{x,y} p(x,y)\log p(x|y) = H(X,Y) - H(Y)\) is the remaining uncertainty in X after observing Y; it is always \(\leq H(X)\) with equality iff \(X \perp Y\). Mutual information \(I(X;Y) = H(X) - H(X|Y) = H(X)+H(Y)-H(X,Y)\) measures statistical dependence—zero iff independent. Equivalently, \(I(X;Y) = D_{\mathrm{KL}}(p(x,y)\|p(x)p(y))\) measures deviation from the independence product. Conditional mutual information \(I(X;Y|Z) = H(X|Z) - H(X|Y,Z)\) is zero iff \(X \perp Y|Z\)—used in graphical models and feature selection. The data processing inequality states \(X \to Y \to Z\) implies \(I(X;Z) \leq I(X;Y)\)—processing can't add information. Shannon's source coding theorem sets the minimum average code length at \(H(X)\) bits; Huffman coding achieves within 1 bit \(\bar{L}_{\mathrm{Huffman}} < H(X) + 1\). Channel capacity \(C = \max_{p(x)} I(X;Y)\) is the maximum reliable transmission rate. Pointwise mutual information \(\mathrm{PMI}(x,y) = \log(p(x,y)/p(x)p(y))\) measures word co-occurrence (foundation of word2vec's PPMI matrices). Normalized mutual information \(\mathrm{NMI}(X,Y) = I(X;Y)/\sqrt{H(X)H(Y)} \in [0,1]\) suits comparing different variable pairs (clustering evaluation). The information bottleneck principle \(\min I(X;Z) - \beta I(Z;Y)\) seeks minimal sufficient statistics.
KL divergence \(D_{\mathrm{KL}}(P\|Q) = \sum_x p(x)\log(p(x)/q(x))\) measures the cost of using Q to encode samples from P. It is non-negative (by Jensen's inequality) and zero iff \(P = Q\), but asymmetric: \(D_{\mathrm{KL}}(P\|Q) \neq D_{\mathrm{KL}}(Q\|P)\). Forward KL (P‖Q) is mean-seeking (Q must cover all of P); reverse KL (Q‖P) is mode-seeking (Q concentrates on P's highest modes—used in VAEs, \(\mathcal{L}_{\mathrm{VAE}} = \mathbb{E}_{q_\phi}[\log p_\theta(x|z)] - D_{\mathrm{KL}}(q_\phi(z|x)\|p(z))\)). The closed-form for two Gaussians is \(\log(\sigma_2/\sigma_1) + (\sigma_1^2+(\mu_1-\mu_2)^2)/(2\sigma_2^2) - \frac{1}{2}\); in VAEs with \(q(z|x) = \mathcal{N}(\mu,\sigma^2)\) and \(p(z) = \mathcal{N}(0,I)\), the KL becomes \(-\frac{1}{2}\sum_j(1+\log\sigma_j^2 - \mu_j^2 - \sigma_j^2)\). The Jensen-Shannon divergence \(\mathrm{JSD}(P\|Q) = \frac{1}{2}D_{\mathrm{KL}}(P\|M) + \frac{1}{2}D_{\mathrm{KL}}(Q\|M)\) with \(M = (P+Q)/2\) is symmetric and bounded by \(\log 2\); the original GAN training minimizes JSD \(\min_G \max_D V(D,G) \equiv -\log 4 + 2\mathrm{JSD}(p\|q)\). Cross-entropy \(H(P,Q) = -\sum_x p(x)\log q(x)\) is the expected code length under Q when sampling from P, always \(\geq H(P)\). Cross-entropy decomposes as \(H(P,Q) = H(P) + D_{\mathrm{KL}}(P\|Q)\)—for one-hot labels, \(H(P) = 0\), so cross-entropy loss exactly equals KL divergence, making MLE equivalent to minimizing cross-entropy and KL. For multi-class classification \(\mathcal{L} = -\sum_k y_k \log \hat y_k\); with softmax outputs, gradient simplifies to \(\hat y - y\). Binary cross-entropy \(\mathcal{L} = -[y\log\hat y + (1-y)\log(1-\hat y)]\) likewise has gradient \(\hat y - y\). Perplexity \(\mathrm{PPL} = 2^{H(P,Q)}\) is the effective vocabulary size in language models. Label smoothing replaces hard labels with \(y_k^{\mathrm{smooth}} = (1-\epsilon)y_k + \epsilon/K\), preventing overconfidence. The evidence lower bound (ELBO) \(\log p(x) \geq \mathbb{E}_q[\log p(x,z)] - \mathbb{E}_q[\log q(z)]\) bounds the log-evidence from below, maximized in variational inference. The reparameterization trick \(\mathbf{z} = \mu + \sigma \odot \boldsymbol{\epsilon}\) makes sampling differentiable for VAEs.