Linear regression models a continuous target as a linear combination of features, predicting \( \hat{y} = w^\top x + b \). It is interpretable, fast, and a baseline against which more elaborate models are measured. Logistic regression adapts the same linear idea to classification by passing the linear score through the logistic (sigmoid) function to produce a class probability, then training via maximum likelihood. Despite their names, both are linear models, and both are surprisingly strong baselines on many problems.
Decision trees take a different tack: they recursively split the data on feature values, creating a tree of if-then rules. Each split is chosen to reduce impurity, measured by entropy (the reduction is called information gain) or by Gini impurity in the CART family. Trees are highly interpretable and handle mixed data types, but a single tree tends to overfit. Ensemble methods fix this. A random forest trains many decision trees on bootstrap samples of the data and averages their predictions — a strategy called bagging, short for bootstrap aggregating. Boosting takes the opposite approach: it trains models sequentially, with each new model focusing on the errors of the previous ones. Gradient boosting fits each new model to the negative gradient (residuals) of the loss, and efficient implementations such as XGBoost, LightGBM, and CatBoost (which handles categorical features natively) have made boosting the dominant approach for tabular data.
Several other classical algorithms remain widely used. K-nearest neighbors (KNN) is a non-parametric method that classifies a new point by the majority label of its closest training examples, with the distance metric doing much of the work. Naive Bayes is a probabilistic classifier that applies Bayes' theorem under a strong, often violated assumption of feature independence; remarkably, it still performs well in text classification and other high-dimensional problems. Support vector machines (SVMs) find the hyperplane that maximally separates classes, and the kernel trick implicitly maps data into higher-dimensional spaces where a linear separator may exist, enabling nonlinear boundaries without computing the mapping explicitly.
Choosing among these algorithms depends on the data size, dimensionality, interpretability requirements, and whether the task is regression or classification. In practice, gradient-boosted trees, logistic regression, and random forests are common starting points on tabular data, while KNN, naive Bayes, and SVMs retain niches in text, small-data, or geometrically structured problems.