A single neuron computes \(a = \sigma(\mathbf{w}^\top\mathbf{x} + b)\) where \(\mathbf{w}\) are weights, \(b\) is bias, and \(\sigma\) is a nonlinear activation. A layer in matrix form: \(\mathbf{z} = W\mathbf{x} + \mathbf{b}\), \(\mathbf{a} = \sigma(\mathbf{z})\), with \(W \in \mathbb{R}^{d_{\text{out}}\times d_{\text{in}}}\) containing \(d_{\text{in}} \times d_{\text{out}} + d_{\text{out}}\) parameters. The forward pass of a multi-layer network stacks such layers: \(\mathbf{a}^{(0)} = \mathbf{x}\), \(\mathbf{z}^{(l)} = W^{(l)}\mathbf{a}^{(l-1)}+\mathbf{b}^{(l)}\), \(\mathbf{a}^{(l)} = \sigma(\mathbf{z}^{(l)})\). Depth allows hierarchical feature extraction while width controls per-layer capacity. Activation functions inject nonlinearity—without them, stacking layers is equivalent to one linear layer. Sigmoid \(\sigma(x) = 1/(1+e^{-x})\) squashes to (0,1) but saturates at ±∞ causing vanishing gradients (max derivative 0.25 at x=0). Tanh \((e^x-e^{-x})/(e^x+e^{-x})\) is zero-centered (better gradient flow) but still saturates; derivative \(1 - \tanh^2(x)\). ReLU \(\max(0,x)\) is the default for hidden layers—no saturation for positive inputs (gradient = 1), cheap to compute—but suffers from dying ReLU if pre-activation stays negative. Leaky ReLU uses slope \(\alpha\) for negative inputs (\(\alpha \approx 0.01\)); PReLU learns \(\alpha\). ELU \(\alpha(e^x-1)\) for \(x \leq 0\) saturates smoothly. GELU \(x\Phi(x)\) weights input by probability of being positive under a Gaussian (used in BERT, GPT). Swish/SiLU \(x\sigma(x)\) is smooth, non-monotonic; SwiGLU \(\mathrm{Swish}(\mathbf{x}_1) \odot \mathbf{x}_2\) is used in Llama's FFN. Softmax \(\sigma(\mathbf{z})_i = e^{z_i}/\sum_j e^{z_j}\) produces a probability distribution (numerically stable version subtracts \(\max_j z_j\) before exponentiation); temperature scaling \(\sigma(\mathbf{z}/T)\) sharpens or flattens.
Backpropagation efficiently computes gradients via the chain rule. The error signal propagates as \(\delta^{(l)} = ((W^{(l+1)})^\top \delta^{(l+1)}) \odot \sigma'(z^{(l)})\), and weight gradients are outer products \(\partial\mathcal{L}/\partial W^{(l)} = \delta^{(l)}(a^{(l-1)})^\top\); bias gradients are simply \(\delta^{(l)}\). Backward cost \(\approx 2\times\) forward cost. Key gradient identities: \(\partial\|x\|_2^2/\partial x = 2x\), \(\partial\mathcal{L}/\partial z = \hat y - y\) for both MSE \(\mathcal{L}_{\mathrm{MSE}} = \frac{1}{N}\sum_i (y_i - \hat y_i)^2\) and cross-entropy + softmax \(\partial z = \hat y - y\). Loss functions include MAE \(\frac{1}{N}\sum |y_i - \hat y_i|\) (robust to outliers, non-differentiable at 0), Huber (quadratic for small residuals, linear for large), and hinge \(\max(0, 1-y\hat f)\) for SVMs. The vanishing gradient problem arises because \(\partial\mathcal{L}/\partial W^{(1)} = \prod_{l=1}^{L-1}(W^{(l)})^\top \cdot \delta^{(L)}\)—if each factor has norm \(<1\), products decay exponentially. Exploding gradients occur analogously when norms exceed 1. Solutions include residual connections \(\mathbf{y} = F(\mathbf{x},\{W_i\}) + \mathbf{x}\) (gradient flows through identity shortcut \(\partial\mathcal{L}/\partial\mathbf{x} = \partial\mathcal{L}/\partial\mathbf{y}(1 + \partial F/\partial\mathbf{x})\)), proper initialization, and normalization layers.
Weight initialization matters for signal propagation. Xavier (Glorot) initialization \(W \sim \mathcal{N}(0, 2/(n_{\text{in}}+n_{\text{out}}))\) preserves variance for tanh/sigmoid activations. He initialization \(W \sim \mathcal{N}(0, 2/n_{\text{in}})\) accounts for ReLU killing half the activations. Normalization layers further stabilize training. Batch normalization normalizes over the batch dimension \(\hat x_i = (x_i - \mu_{\mathcal{B}})/\sqrt{\sigma_{\mathcal{B}}^2+\epsilon}\), then rescales with learned \(\gamma, \beta\); at inference, uses running exponential moving averages \(\mu_{\mathrm{running}} = (1-m)\mu_{\mathrm{running}} + m\mu_{\mathcal{B}}\). Layer normalization normalizes over features within each sample (batch-independent, preferred for transformers). Group normalization divides channels into groups; instance normalization is per-sample per-channel (used in style transfer). Spectral normalization \(\bar W = W/\sigma_{\max}(W)\) enforces 1-Lipschitz layers (stabilizes GANs). Regularization prevents overfitting: L2 (weight decay) adds \(\frac{\lambda}{2}\sum w_j^2\) with gradient \(\lambda\theta\) (equivalent to Gaussian prior in MAP); L1 adds \(\lambda\sum|w_j|\) (Laplace prior, encourages sparsity). Dropout randomly zeroes activations \(\tilde a_i = a_i m_i/(1-p)\) with Bernoulli mask \(m_i\); inverted dropout scaling \(1/(1-p)\) makes \(\mathbb{E}[\tilde a] = a\), so no rescaling needed at test time—acts as ensemble training.
Convolutional networks introduce strong inductive biases. A 2D convolution slides a kernel over the input: \((I*K)[i,j] = \sum_{mn} I[i+m,j+n]K[m,n]\)—local connectivity and weight sharing (translation equivariance, parameter-independent of spatial size). Pooling reduces spatial dimensions: max pooling \(\max_{(m,n)} X_{im,jn}\) takes the maximum in each window, average pooling takes the mean. Receptive field grows linearly with depth (kernel size \(k\) gives \(k + (k-1)(l-1)\) at layer \(l\)). Residual (skip) connections \(\mathbf{y} = F(\mathbf{x},\{W_i\}) + \mathbf{x}\) let layers learn residuals; the identity shortcut guarantees non-vanishing gradients. Pre-activation design moves BN+ReLU before weights, giving a clean identity path through the skip connection. The universal approximation theorem guarantees that a single hidden layer with enough neurons can approximate any continuous function \(\sup|\hat f - f| < \epsilon\) to arbitrary precision, but doesn't address learnability or parameter efficiency—deep networks with ReLU create \(O((N/k)^{(k-1)d})\) linear regions, exponentially more than wide shallow networks with the same parameter count. Transposed convolutions provide learnable upsampling in decoders. Spectral norm regularization, knowledge distillation \(\mathcal{L}_{\mathrm{KD}} = T^2 D_{\mathrm{KL}}(\text{softmax}(z_T/T)\|\text{softmax}(z_S/T))\) with temperature \(T\), and the LSTM cell state \(C_t = f_t \odot C_{t-1} + i_t \odot \tilde C_t\) (forget, input, output gates \(f_t, i_t, o_t = \sigma(W[h,x]+b)\) with candidate \(\tilde C_t = \tanh(W_C[h,x]+b_C)\)) are additional architectural primitives. Sharpness-aware minimization (SAM) \(\min_\theta \max_{\|\epsilon\|\leq\rho} \mathcal{L}(\theta+\epsilon)\) explicitly seeks flat minima for better generalization.