Training a machine learning model requires defining an objective and a procedure to minimize it. The loss function quantifies how far predictions are from the true targets; common choices include Mean Squared Error \( \text{MSE} = \frac{1}{n} \sum (y_{\text{pred}} - y_{\text{actual}})^2 \) for regression and cross-entropy \( L = -[y \log p + (1-y) \log (1-p)] \) for classification. Gradient descent minimizes the loss by iteratively updating parameters in the direction of the negative gradient. The learning rate controls the step size—too large causes overshooting, while too small leads to slow convergence. Variants include batch, stochastic (SGD), and mini-batch gradient descent. SGD with momentum adds a fraction of previous updates to accelerate convergence through ravines, while Adam combines momentum with RMSProp-style adaptive per-parameter learning rates and works well with default settings such as learning rate \( 10^{-3} \). Learning rate schedules such as step decay, exponential decay, and cosine annealing often improve convergence over a fixed learning rate.
A central concern in machine learning is balancing model complexity. Overfitting occurs when a model learns noise and details of the training data, yielding high training accuracy but poor test performance. Underfitting is the opposite: the model is too simple to capture underlying patterns and performs poorly on both sets. The bias-variance tradeoff formalizes this tension—bias from overly simple assumptions and variance from sensitivity to training data fluctuations—and the goal is to find a complexity that minimizes total error. Remedies include regularization, more training data, and adjusting model capacity. Regularization adds a penalty to the loss function: L1 (Lasso) drives some weights to zero for sparse models, while L2 (Ridge) shrinks weights smoothly. Elastic Net combines both. In neural networks, dropout randomly zeroes neurons during training, label smoothing prevents overconfident outputs, and early stopping halts training when validation performance plateaus.
Evaluating models reliably requires holding out data the model never sees. A train/test split divides the dataset, commonly in 80/20 or 70/30 proportions, and a separate validation set helps tune hyperparameters. K-fold cross-validation rotates folds to produce a more robust estimate of generalization; scikit-learn provides this via cross_val_score with cv=5. Hyperparameter tuning searches the configuration space using grid search, random search, or Bayesian optimization, with GridSearchCV and RandomizedSearchCV as convenient implementations.
For classification, the confusion matrix summarizes true and false positives and negatives, from which accuracy \( (\text{TP} + \text{TN}) / N \), precision \( \text{TP} / (\text{TP} + \text{FP}) \), recall \( \text{TP} / (\text{TP} + \text{FN}) \), and the F1-score—the harmonic mean of precision and recall—are derived. Accuracy can mislead on imbalanced datasets, where precision, recall, F1, Cohen's Kappa \( \kappa = (p_o - p_e) / (1 - p_e) \), or the ROC curve and its AUC are more informative. The ROC curve plots true positive rate against false positive rate across thresholds, and AUC summarizes it as a single number where 0.5 is random and 1.0 is perfect. For regression, MAE measures average absolute error and is robust to outliers, while MSE and RMSE penalize large errors more heavily. The R² metric measures the proportion of variance explained, and adjusted R² penalizes adding irrelevant features. Log loss is the standard objective for probabilistic classifiers, perplexity summarizes language model uncertainty, and BLEU and ROUGE evaluate text generation against references, with BLEU emphasizing precision and ROUGE emphasizing recall. Learning curves plotting training and validation metrics over time help diagnose bias and variance issues visually.