To estimate how a model will perform on unseen data, practitioners split the data into training and testing sets. This simple train-test split protects against overfitting, the failure mode in which a model performs well on training data but poorly on new data. A more reliable estimate comes from cross-validation, where the data is divided into folds and each fold is used in turn as the test set while the others train the model. K-fold cross-validation with \(k\) folds averages the result across all \(k\) test runs. When class proportions are important, stratified k-fold preserves those proportions in every fold, giving more stable estimates under imbalance.
Closely related to overfitting is data leakage, the contamination of training by information that would not be available at prediction time — for example, imputing missing values using statistics computed on the full dataset, or peeking at test labels during feature engineering. Leakage inflates performance estimates and produces models that collapse in production. Underfitting is the opposite failure: the model is too simple to capture the patterns in the data. The bias-variance tradeoff describes the tension between underfitting (high bias, low variance) and overfitting (low bias, high variance), and most modeling choices are implicitly attempts to balance the two. Regularization is one such mechanism: it adds a penalty for complexity to the loss function. L1 regularization (Lasso) penalizes the absolute values of coefficients and pushes some to zero, producing sparse models, while L2 regularization (Ridge) penalizes squared coefficients and shrinks them smoothly. Elastic Net combines L1 and L2.
Training a model usually means minimizing a loss function that quantifies how wrong predictions are. For regression, common losses include mean squared error (MSE), the average of squared differences between predictions and actuals, and mean absolute error (MAE), the average of absolute differences. For classification, cross-entropy loss measures the distance between predicted probabilities and the true distribution. The optimizer that drives this minimization is typically gradient descent, which iteratively updates parameters in the direction opposite the gradient of the loss. The learning rate controls the step size of each update; too large and the model diverges, too small and training crawls. Stochastic gradient descent uses single examples or small batches (mini-batch gradient descent) to make updates cheap and noisy enough to escape poor local minima, at the cost of a less stable trajectory.
Beyond the loss, classification models are evaluated with metrics tailored to the problem. Accuracy is the fraction of correct predictions, but it can mislead on imbalanced data; precision (\( \text{TP} / (\text{TP}+\text{FP}) \)) measures how many of the predicted positives are real, while recall (\( \text{TP} / (\text{TP}+\text{FN}) \)) measures how many of the actual positives were caught. The F1 score is the harmonic mean of precision and recall, useful when both matter. The confusion matrix lays out predicted versus actual counts for each class, supporting derived metrics. Threshold-based summaries visualize the trade-off: the ROC curve plots true positive rate against false positive rate at various thresholds and the AUC summarizes discrimination as a single number; the precision-recall curve is more informative when the positive class is rare. Statistical significance, hypothesis testing, and p-values complement these metrics by helping judge whether observed differences are likely due to chance under a null hypothesis.