Neural networks are computational models inspired by biological neurons. The simplest unit is the perceptron, which computes a weighted sum of inputs, adds a bias, and applies a step function. A single perceptron can only solve linearly separable problems like AND and OR, but cannot learn XOR. Stacking perceptrons in layers produces a multilayer perceptron (MLP)—a fully connected feedforward network with one or more hidden layers. Each connection carries a weight that is adjusted during training, and MLPs trained with backpropagation form the foundation of deep learning.
Training a neural network requires computing gradients of the loss with respect to each weight. Backpropagation applies the chain rule of calculus to propagate gradients backward from the output layer to earlier layers, after which an optimizer updates the weights. Activation functions introduce the non-linearity that lets networks learn complex patterns. The sigmoid function maps any input to a value in [0, 1] and is used in logistic regression and binary output layers, but suffers from the vanishing gradient problem for very large or small inputs. ReLU, defined as \( f(x) = \max(0, x) \), is the most widely used hidden-layer activation because it mitigates vanishing gradients and enables fast training; Leaky ReLU allows a small negative slope. Tanh maps to [-1, 1] and is centered at zero, while softmax \( \text{softmax}(z_i) = e^{z_i} / \sum e^{z_j} \) converts a vector of logits into a probability distribution used in multi-class output layers.
Deep networks face two well-known training pathologies. The vanishing gradient problem occurs when gradients become extremely small in early layers, slowing or halting learning, and is common with sigmoid and tanh activations and in RNNs. Solutions include ReLU activations, batch normalization, residual connections, and LSTM/GRU architectures. The exploding gradient problem, where gradients grow exponentially, can be controlled with gradient clipping, which caps gradient norms at a threshold. Proper weight initialization is also crucial: Xavier (Glorot) initialization sets weight variance to \( 2 / (\text{fan\_in} + \text{fan\_out}) \) and suits tanh and sigmoid activations, while He (Kaiming) initialization uses \( 2 / \text{fan\_in} \) and is tailored for ReLU. Batch normalization normalizes layer inputs across a mini-batch and allows higher learning rates, while layer normalization normalizes across features within a single sample, making it ideal for transformers and RNNs.
Regularization in deep learning takes several forms. Dropout randomly zeroes a fraction of neuron outputs during training, forcing the network to learn robust, redundant features; typical dropout rates are 0.2 to 0.5 and dropout is disabled during inference. Weight decay (L2 regularization) penalizes large weights, and label smoothing prevents overconfident outputs. Early stopping halts training when validation performance ceases to improve. Residual learning, introduced in ResNet, adds skip connections that let layers learn identity mappings via \( \text{output} = F(x) + x \), enabling training of networks hundreds of layers deep, and won the 2015 ImageNet competition. Together, these techniques make training deep networks stable, efficient, and generalizable.