When labels are absent, unsupervised methods extract structure directly from the data. Clustering algorithms group similar points: k-means partitions data into \(k\) clusters by minimizing within-cluster variance, hierarchical clustering builds a tree of nested groups by successive merging (agglomerative) or splitting (divisive), and DBSCAN finds clusters as dense regions separated by sparse noise, which lets it discover clusters of arbitrary shape and identify outliers. Choosing \(k\) in k-means is often guided by the elbow method, which plots the within-cluster cost against \(k\) and looks for a bend, and by the silhouette score, which measures how similar each point is to its own cluster compared to neighboring clusters.
Dimensionality reduction compresses many features into fewer while preserving important structure. Principal Component Analysis (PCA) finds orthogonal directions of maximum variance, producing a linear, interpretable embedding. For visualization and structure discovery in high dimensions, nonlinear methods such as t-SNE and UMAP are popular; UMAP in particular is fast, scalable, and tends to preserve more global structure than t-SNE. These techniques are invaluable for both exploration and as preprocessing for downstream models.
Deep learning extends the same learning paradigm with flexible, layered models. A neural network is composed of layers of interconnected nodes, or neurons; the simplest is the perceptron, a single linear classifier. A multilayer perceptron (MLP) stacks fully connected hidden layers between input and output. Each neuron applies an activation function to its weighted sum. ReLU, defined as \(\max(0, x)\), is the default choice for hidden layers due to its well-behaved gradients; sigmoid, \(1/(1+e^{-x})\), squashes outputs to \((0,1)\) and is used for binary probabilities; and softmax converts a vector of logits into a probability distribution over classes. Training these networks is driven by backpropagation, an efficient algorithm that computes gradients of the loss with respect to each parameter by applying the chain rule from the output back through the layers.
Specialized architectures target specific data modalities. Convolutional neural networks (CNNs) use convolutional layers to exploit spatial structure in grid-like data such as images, making them the workhorse of computer vision. Recurrent neural networks (RNNs) handle sequences by passing a hidden state between timesteps; LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) address the difficulty RNNs have capturing long-range dependencies through gating mechanisms. Transformers replace recurrence with self-attention, a mechanism that lets each position in a sequence attend to all others, and they are the foundation of modern NLP. BERT uses a bidirectional transformer encoder for representation learning, while GPT uses an autoregressive transformer decoder for generation. Adapting these large pretrained models is rarely done from scratch: transfer learning reuses a model trained on one task for another related task, and fine-tuning continues its training on task-specific data to specialize it. Training is stabilized and regularized by techniques such as dropout (randomly disabling neurons during training), batch normalization (normalizing layer inputs to accelerate and stabilize training), early stopping (halting when validation performance plateaus), and data augmentation (creating new training examples by transforming existing ones, especially important in vision).