Different data modalities call for specialized architectures. Convolutional Neural Networks (CNNs) are designed for spatial data such as images and audio spectrograms. A convolutional layer applies learnable filters (kernels) that slide across the input, sharing weights across spatial positions and drastically reducing parameters compared to dense layers; common kernel sizes are 3×3 and 5×5. Pooling layers, typically max pooling with 2×2 windows, downsample feature maps to reduce computation and overfitting; modern architectures sometimes use strided convolutions instead. The receptive field is the region of input that influences a particular neuron; deeper neurons have larger receptive fields, capturing more global context. Fully connected layers at the end of a CNN combine learned features for final prediction.
Several CNN-based architectures address specific computer vision tasks. Semantic segmentation models like U-Net use a symmetric encoder–decoder structure with skip connections that pass high-resolution features from encoder to decoder, originally designed for biomedical image segmentation and now widely used for dense per-pixel prediction. Mask R-CNN extends Faster R-CNN with a parallel branch that predicts a binary mask for each detected object, enabling instance segmentation that distinguishes individual object outlines. YOLO (You Only Look Once) reframes object detection as a single regression problem, predicting bounding boxes and class probabilities in one forward pass and achieving real-time performance for applications in autonomous driving, robotics, and surveillance.
Sequential data such as text and time series motivates Recurrent Neural Networks (RNNs), which maintain hidden states that act as memory across timesteps. Vanilla RNNs suffer from vanishing and exploding gradients, making them hard to train over long sequences. Long Short-Term Memory (LSTM) networks address this with gating mechanisms—an input gate, forget gate, and output gate—that selectively remember or forget information, along with a cell state that carries information across long distances. GRUs (Gated Recurrent Units) simplify LSTMs by combining the forget and input gates into a single update gate plus a reset gate, achieving comparable performance with fewer parameters. LSTMs and GRUs power applications in speech recognition, machine translation, and time-series forecasting.
The transformer architecture, based entirely on self-attention, has largely supplanted RNNs for sequence modeling. Self-attention computes a weighted sum over all tokens in a sequence using \( \text{Attention}(Q, K, V) = \text{softmax}(Q K^\top / \sqrt{d}) V \), where weights are derived from learnable query, key, and value projections. Multi-head attention runs several self-attention operations in parallel, each focusing on different representation subspaces, and concatenates their outputs. Because self-attention is permutation-invariant, positional encodings inject token order information. The encoder-decoder architecture maps an input sequence to an output sequence via an encoder that compresses the input and a decoder that generates tokens conditioned on that context. BERT is a bidirectional encoder pre-trained with masked language modeling and next sentence prediction, while GPT is a decoder-only transformer pre-trained with autoregressive next-token prediction; successive GPT versions have scaled to hundreds of billions of parameters and form the basis for ChatGPT-style assistants. Large generative pre-trained models exhibit in-context learning, performing new tasks from examples given in the prompt without any weight updates, and prompt engineering—including few-shot examples, chain-of-thought reasoning, and role assignment—has become a key skill for working with these models.