Many foundational algorithms form the backbone of practical machine learning. Linear regression models the relationship between input features and a continuous target by fitting a straight line or hyperplane that minimizes the sum of squared residuals; in scikit-learn it is implemented as LinearRegression(). Logistic regression adapts this idea for binary classification by wrapping the linear output in a sigmoid. Decision trees learn hierarchical if-then-else rules from data, splitting on features to partition the input space. While highly interpretable, single trees are prone to overfitting.
Ensemble methods combine many models for stronger predictions. A random forest trains many decision trees on bootstrap samples with random feature subsets, then averages their predictions (regression) or takes a majority vote (classification). This bagging approach reduces variance. Gradient boosting builds trees sequentially, with each new tree fitting the residual errors of the previous ensemble; popular implementations include XGBoost, LightGBM, and CatBoost. In general, bagging primarily reduces variance while boosting primarily reduces bias, and both can be used to combine heterogeneous models via stacking with a meta-learner.
Beyond trees and ensembles, several other classical algorithms are widely used. Support Vector Machines find the optimal hyperplane that maximizes the margin between classes, with support vectors defining the boundary. The kernel trick allows SVMs to find non-linear boundaries via kernels such as RBF, polynomial, or sigmoid. k-Nearest Neighbors (KNN) is a lazy learner that classifies points based on the majority class among their \( k \) nearest neighbors using distance metrics like Euclidean or Manhattan. Naive Bayes is a probabilistic classifier based on Bayes' theorem with an independence assumption that, despite its naivety, often works surprisingly well for text classification and spam filtering; variants include Multinomial, Bernoulli, and Gaussian Naive Bayes.
For unsupervised tasks, k-means partitions data into \( k \) clusters by iteratively assigning points to the nearest centroid and updating centroids to the cluster means; the elbow method helps choose \( k \) by looking for the inflection point in the within-cluster sum of squares. DBSCAN groups density-connected points and flags low-density regions as outliers, allowing arbitrarily shaped clusters without specifying \( k \) in advance. Gaussian Mixture Models generalize k-means by allowing soft, elliptical clusters fitted via the Expectation-Maximization algorithm. Hierarchical clustering builds a tree of clusters called a dendrogram, useful when the number of clusters is unknown but expensive at scale. Dimensionality reduction techniques such as PCA, t-SNE, and UMAP project high-dimensional data into fewer dimensions, with t-SNE and UMAP being particularly popular for visualization while UMAP often preserves more global structure. For text data, classical representations include Bag-of-Words, which counts word occurrences, and TF-IDF, which downweights common words across documents.