Classification algorithms can be organized by how they treat the joint distribution of features and labels. Generative classifiers model the joint \(p(x, y)\) by specifying \(p(y)\) and \(p(x \mid y)\); discriminative classifiers model \(p(y \mid x)\) directly. Naive Bayes is the canonical generative model: assuming features are conditionally independent given the class label \(y \sim\) Categorical, the joint becomes a product of per-feature likelihoods. A Bernoulli Naive Bayes uses Bernoulli(\(p_{c,i}\)) for each feature given class \(c\), while a Gaussian Naive Bayes uses \(N(x_i; \mu_{c,i}, \sigma^2_{c,i})\). Predictions are made by the maximum posterior class. Gaussian Discriminant Analysis (GDA) is a richer generative model with full multivariate Gaussian class-conditionals \(p(x \mid y = k) = N(x; \mu_k, \Sigma_k)\) and class priors \(p(y = k) = \pi_k\); the Bayes-optimal decision boundary is quadratic. Adding the assumption that all classes share a common covariance \(\Sigma\) turns GDA into Linear Discriminant Analysis (LDA), with a linear decision boundary, while removing that assumption gives Quadratic Discriminant Analysis (QDA) with a quadratic boundary. LDA is closely related to Fisher's linear discriminant, since both seek a projection that maximizes class separability — Fisher's criterion yields a closed-form projection, while LDA arises from the equal-covariance generative assumption.
Discriminative classification typically uses a softmax or sigmoid parameterization. The softmax function maps real-valued scores to a Categorical distribution via \(\text{softmax}(z_i) = e^{z_i} / \sum_j e^{z_j}\); it arises naturally from the multinomial logistic regression likelihood. The sigmoid \(\sigma(z) = 1/(1 + e^{-z})\) maps a real number to \((0, 1)\) and parameterizes Bernoulli probabilities in logistic regression. The logistic (sigmoid) distribution is the corresponding continuous distribution whose CDF equals \(\sigma\), with PDF \(f(x) = e^{-(x - \mu)/s} / (s(1 + e^{-(x - \mu)/s})^2)\), mean \(\mu\), and variance \(s^2 \pi^2 / 3\). The softmax itself can be viewed as a Gibbs distribution over discrete states with energies \(-z_i\) and temperature 1, and softmax with temperature \(T\) corresponds to \(\exp(z_i / T)\).
The corresponding loss functions are negative log-likelihoods: categorical cross-entropy \(-\sum_n \sum_k y_{n,k} \log p(y_{n,k} \mid x_n)\) for \(K\)-class softmax classifiers, binary cross-entropy \(-\frac{1}{N}\sum_n [y_n \log p(x_n) + (1 - y_n) \log(1 - p(x_n))]\) for sigmoid-parameterized Bernoullis, and MSE for a Gaussian with fixed variance up to a constant. The choice between discriminative and generative approaches reflects modeling priorities: generative classifiers can synthesize new examples and incorporate prior knowledge about how \(x\) is generated, while discriminative classifiers often achieve higher accuracy when the goal is purely prediction and the conditional \(p(y \mid x)\) is all that matters. In both cases, the parameterizations (softmax, sigmoid, multivariate Gaussian) and the associated losses connect directly back to the underlying probability distributions.