Machine Learning Programming

building blocks of machine learning, starting with the broad distinction between supervised and unsupervised learning and moving into the specific algorithms that power each approach. You'll review the differences between classification and regression, learn how methods like linear regression, logistic regression, decision trees, random forests, and Support Vector Machines work, and explore clustering with k-means. Alongside the algorithms, the cards cover key ideas you'll encounter again and again — gradient descent, overfitting, underfitting, cross-validation, and feature engineering — as well as scikit-learn, one of the most popular Python libraries for putting these techniques into practice.

It's a great fit if you're a beginner stepping into machine learning, a programming student preparing for coursework or interviews, or a developer who wants a clear, jargon-light refresher on the fundamentals. The questions are phrased as short definitions, so the deck works well whether you're starting from scratch or just solidifying concepts you already half-remember.

Because many of these terms are closely related — overfitting and underfitting, classification and regression, supervised and unsupervised — try to answer each card in your own words before flipping it over. Reviewing a few cards at a time across several days is far more effective than cramming everything in one session, since machine learning vocabulary tends to stick when you encounter it in spaced bursts. When you're ready, pair the flashcards with a small coding exercise in scikit-learn to see how the concepts connect in real code.

Machine learning is broadly categorized by how learning signals are provided to the model. In supervised learning, models are trained on labeled data—each example pairs an input with a known correct output—and learn to predict labels for new inputs. Common supervised tasks include classification, which predicts discrete labels such as spam versus not spam, and regression, which predicts continuous values like house prices. In contrast, unsupervised learning trains on unlabeled data and requires the model to discover hidden structure such as clusters, low-dimensional manifolds, or outliers. Techniques such as k-means clustering and PCA fall into this category.

Beyond these two main paradigms, several other learning settings are widely used. Semi-supervised learning combines a small amount of labeled data with a large pool of unlabeled data, which is useful when labeling is expensive. Self-supervised learning goes further by generating its own supervision signal from the input—for example, predicting masked words or contrasting image transformations—and powers modern foundation models such as BERT and CLIP. Reinforcement learning involves an agent learning a policy by interacting with an environment to maximize a cumulative reward signal, as seen in game-playing AIs and robotics. Related paradigms include multi-task learning, where one model shares representations across several tasks, and transfer learning, where a pre-trained model is fine-tuned for a new but related task.

A practical distinction within supervised learning is between classification and regression. Classification outputs a category from a discrete set of classes, while regression outputs a real number. Logistic regression, despite its name, is a classification algorithm that uses the sigmoid function \( \sigma(x) = \frac{1}{1 + e^{-x}} \) to map a linear score into a probability between 0 and 1, applying a threshold (often 0.5) to assign a class label. Understanding which learning paradigm and task type fits a problem is the first step in any machine learning project.

Many foundational algorithms form the backbone of practical machine learning. Linear regression models the relationship between input features and a continuous target by fitting a straight line or hyperplane that minimizes the sum of squared residuals; in scikit-learn it is implemented as LinearRegression(). Logistic regression adapts this idea for binary classification by wrapping the linear output in a sigmoid. Decision trees learn hierarchical if-then-else rules from data, splitting on features to partition the input space. While highly interpretable, single trees are prone to overfitting.

Ensemble methods combine many models for stronger predictions. A random forest trains many decision trees on bootstrap samples with random feature subsets, then averages their predictions (regression) or takes a majority vote (classification). This bagging approach reduces variance. Gradient boosting builds trees sequentially, with each new tree fitting the residual errors of the previous ensemble; popular implementations include XGBoost, LightGBM, and CatBoost. In general, bagging primarily reduces variance while boosting primarily reduces bias, and both can be used to combine heterogeneous models via stacking with a meta-learner.

Beyond trees and ensembles, several other classical algorithms are widely used. Support Vector Machines find the optimal hyperplane that maximizes the margin between classes, with support vectors defining the boundary. The kernel trick allows SVMs to find non-linear boundaries via kernels such as RBF, polynomial, or sigmoid. k-Nearest Neighbors (KNN) is a lazy learner that classifies points based on the majority class among their \( k \) nearest neighbors using distance metrics like Euclidean or Manhattan. Naive Bayes is a probabilistic classifier based on Bayes' theorem with an independence assumption that, despite its naivety, often works surprisingly well for text classification and spam filtering; variants include Multinomial, Bernoulli, and Gaussian Naive Bayes.

For unsupervised tasks, k-means partitions data into \( k \) clusters by iteratively assigning points to the nearest centroid and updating centroids to the cluster means; the elbow method helps choose \( k \) by looking for the inflection point in the within-cluster sum of squares. DBSCAN groups density-connected points and flags low-density regions as outliers, allowing arbitrarily shaped clusters without specifying \( k \) in advance. Gaussian Mixture Models generalize k-means by allowing soft, elliptical clusters fitted via the Expectation-Maximization algorithm. Hierarchical clustering builds a tree of clusters called a dendrogram, useful when the number of clusters is unknown but expensive at scale. Dimensionality reduction techniques such as PCA, t-SNE, and UMAP project high-dimensional data into fewer dimensions, with t-SNE and UMAP being particularly popular for visualization while UMAP often preserves more global structure. For text data, classical representations include Bag-of-Words, which counts word occurrences, and TF-IDF, which downweights common words across documents.

Training a machine learning model requires defining an objective and a procedure to minimize it. The loss function quantifies how far predictions are from the true targets; common choices include Mean Squared Error \( \text{MSE} = \frac{1}{n} \sum (y_{\text{pred}} - y_{\text{actual}})^2 \) for regression and cross-entropy \( L = -[y \log p + (1-y) \log (1-p)] \) for classification. Gradient descent minimizes the loss by iteratively updating parameters in the direction of the negative gradient. The learning rate controls the step size—too large causes overshooting, while too small leads to slow convergence. Variants include batch, stochastic (SGD), and mini-batch gradient descent. SGD with momentum adds a fraction of previous updates to accelerate convergence through ravines, while Adam combines momentum with RMSProp-style adaptive per-parameter learning rates and works well with default settings such as learning rate \( 10^{-3} \). Learning rate schedules such as step decay, exponential decay, and cosine annealing often improve convergence over a fixed learning rate.

A central concern in machine learning is balancing model complexity. Overfitting occurs when a model learns noise and details of the training data, yielding high training accuracy but poor test performance. Underfitting is the opposite: the model is too simple to capture underlying patterns and performs poorly on both sets. The bias-variance tradeoff formalizes this tension—bias from overly simple assumptions and variance from sensitivity to training data fluctuations—and the goal is to find a complexity that minimizes total error. Remedies include regularization, more training data, and adjusting model capacity. Regularization adds a penalty to the loss function: L1 (Lasso) drives some weights to zero for sparse models, while L2 (Ridge) shrinks weights smoothly. Elastic Net combines both. In neural networks, dropout randomly zeroes neurons during training, label smoothing prevents overconfident outputs, and early stopping halts training when validation performance plateaus.

Evaluating models reliably requires holding out data the model never sees. A train/test split divides the dataset, commonly in 80/20 or 70/30 proportions, and a separate validation set helps tune hyperparameters. K-fold cross-validation rotates folds to produce a more robust estimate of generalization; scikit-learn provides this via cross_val_score with cv=5. Hyperparameter tuning searches the configuration space using grid search, random search, or Bayesian optimization, with GridSearchCV and RandomizedSearchCV as convenient implementations.

For classification, the confusion matrix summarizes true and false positives and negatives, from which accuracy \( (\text{TP} + \text{TN}) / N \), precision \( \text{TP} / (\text{TP} + \text{FP}) \), recall \( \text{TP} / (\text{TP} + \text{FN}) \), and the F1-score—the harmonic mean of precision and recall—are derived. Accuracy can mislead on imbalanced datasets, where precision, recall, F1, Cohen's Kappa \( \kappa = (p_o - p_e) / (1 - p_e) \), or the ROC curve and its AUC are more informative. The ROC curve plots true positive rate against false positive rate across thresholds, and AUC summarizes it as a single number where 0.5 is random and 1.0 is perfect. For regression, MAE measures average absolute error and is robust to outliers, while MSE and RMSE penalize large errors more heavily. The R² metric measures the proportion of variance explained, and adjusted R² penalizes adding irrelevant features. Log loss is the standard objective for probabilistic classifiers, perplexity summarizes language model uncertainty, and BLEU and ROUGE evaluate text generation against references, with BLEU emphasizing precision and ROUGE emphasizing recall. Learning curves plotting training and validation metrics over time help diagnose bias and variance issues visually.

The quality and form of input features often matter more than algorithm choice. Feature engineering creates, transforms, or selects features to expose useful signal to the model, including extracting date and time components, binning continuous variables, creating interaction features, and encoding categorical variables. One-hot encoding turns each category into a binary column with a 1 indicating the presence of that category, while label encoding assigns integers; scikit-learn provides OneHotEncoder and pandas offers pd.get_dummies for the same purpose. Good feature engineering often has a bigger impact than algorithm choice.

Numerical features often need scaling so that no single feature dominates. Normalization (min-max scaling) rescales features to a fixed range, typically [0, 1] using \( (x - \min) / (\max - \min) \), while standardization (Z-score) centers features at zero mean with unit variance using \( (x - \mu) / \sigma \). Scaling is essential for distance-based algorithms like KNN and SVM, and for gradient-based methods like neural networks and logistic regression, where features with larger magnitudes can dominate learning.

Real datasets have imperfections that must be addressed. Missing data can be removed, imputed with mean, median, or mode, or filled using methods like KNN imputation, with scikit-learn's SimpleImputer providing convenient strategies; indicator variables can flag missingness as an additional feature. Class imbalance, where one class dominates, biases models toward the majority class and can be mitigated by resampling such as oversampling the minority with SMOTE, undersampling the majority, class weighting in the loss, or threshold tuning. Outliers—points that significantly deviate—may be data errors or genuine rare events; robust methods such as Huber loss, RANSAC, or Isolation Forest are less sensitive to them. Data augmentation artificially expands training data via label-preserving transformations such as rotations and flips for images, or synonym replacement and back-translation for text.

Feature selection methods reduce dimensionality, noise, and overfitting. Filter approaches use correlation, chi-square, or mutual information to rank features, with SelectKBest selecting the top-k by mutual information. Wrapper methods like recursive feature elimination (RFE) recursively remove the least important features by retraining a model. Embedded methods such as Lasso or tree importances perform selection as part of training. Scikit-learn's Pipeline object chains transformers and an estimator so the same preprocessing is consistently applied during training and prediction, preventing data leakage. Data leakage, where information from outside the training set bleeds into the model, produces falsely good evaluation scores and is a common cause of deployment failures. The curse of dimensionality describes how data becomes sparse and distances lose meaning in very high dimensions, motivating dimensionality reduction via PCA or UMAP. In production, data drift occurs when input distributions change and concept drift occurs when the relationship between features and target changes, both requiring monitoring and retraining.

Neural networks are computational models inspired by biological neurons. The simplest unit is the perceptron, which computes a weighted sum of inputs, adds a bias, and applies a step function. A single perceptron can only solve linearly separable problems like AND and OR, but cannot learn XOR. Stacking perceptrons in layers produces a multilayer perceptron (MLP)—a fully connected feedforward network with one or more hidden layers. Each connection carries a weight that is adjusted during training, and MLPs trained with backpropagation form the foundation of deep learning.

Training a neural network requires computing gradients of the loss with respect to each weight. Backpropagation applies the chain rule of calculus to propagate gradients backward from the output layer to earlier layers, after which an optimizer updates the weights. Activation functions introduce the non-linearity that lets networks learn complex patterns. The sigmoid function maps any input to a value in [0, 1] and is used in logistic regression and binary output layers, but suffers from the vanishing gradient problem for very large or small inputs. ReLU, defined as \( f(x) = \max(0, x) \), is the most widely used hidden-layer activation because it mitigates vanishing gradients and enables fast training; Leaky ReLU allows a small negative slope. Tanh maps to [-1, 1] and is centered at zero, while softmax \( \text{softmax}(z_i) = e^{z_i} / \sum e^{z_j} \) converts a vector of logits into a probability distribution used in multi-class output layers.

Deep networks face two well-known training pathologies. The vanishing gradient problem occurs when gradients become extremely small in early layers, slowing or halting learning, and is common with sigmoid and tanh activations and in RNNs. Solutions include ReLU activations, batch normalization, residual connections, and LSTM/GRU architectures. The exploding gradient problem, where gradients grow exponentially, can be controlled with gradient clipping, which caps gradient norms at a threshold. Proper weight initialization is also crucial: Xavier (Glorot) initialization sets weight variance to \( 2 / (\text{fan\_in} + \text{fan\_out}) \) and suits tanh and sigmoid activations, while He (Kaiming) initialization uses \( 2 / \text{fan\_in} \) and is tailored for ReLU. Batch normalization normalizes layer inputs across a mini-batch and allows higher learning rates, while layer normalization normalizes across features within a single sample, making it ideal for transformers and RNNs.

Regularization in deep learning takes several forms. Dropout randomly zeroes a fraction of neuron outputs during training, forcing the network to learn robust, redundant features; typical dropout rates are 0.2 to 0.5 and dropout is disabled during inference. Weight decay (L2 regularization) penalizes large weights, and label smoothing prevents overconfident outputs. Early stopping halts training when validation performance ceases to improve. Residual learning, introduced in ResNet, adds skip connections that let layers learn identity mappings via \( \text{output} = F(x) + x \), enabling training of networks hundreds of layers deep, and won the 2015 ImageNet competition. Together, these techniques make training deep networks stable, efficient, and generalizable.

Different data modalities call for specialized architectures. Convolutional Neural Networks (CNNs) are designed for spatial data such as images and audio spectrograms. A convolutional layer applies learnable filters (kernels) that slide across the input, sharing weights across spatial positions and drastically reducing parameters compared to dense layers; common kernel sizes are 3×3 and 5×5. Pooling layers, typically max pooling with 2×2 windows, downsample feature maps to reduce computation and overfitting; modern architectures sometimes use strided convolutions instead. The receptive field is the region of input that influences a particular neuron; deeper neurons have larger receptive fields, capturing more global context. Fully connected layers at the end of a CNN combine learned features for final prediction.

Several CNN-based architectures address specific computer vision tasks. Semantic segmentation models like U-Net use a symmetric encoder–decoder structure with skip connections that pass high-resolution features from encoder to decoder, originally designed for biomedical image segmentation and now widely used for dense per-pixel prediction. Mask R-CNN extends Faster R-CNN with a parallel branch that predicts a binary mask for each detected object, enabling instance segmentation that distinguishes individual object outlines. YOLO (You Only Look Once) reframes object detection as a single regression problem, predicting bounding boxes and class probabilities in one forward pass and achieving real-time performance for applications in autonomous driving, robotics, and surveillance.

Sequential data such as text and time series motivates Recurrent Neural Networks (RNNs), which maintain hidden states that act as memory across timesteps. Vanilla RNNs suffer from vanishing and exploding gradients, making them hard to train over long sequences. Long Short-Term Memory (LSTM) networks address this with gating mechanisms—an input gate, forget gate, and output gate—that selectively remember or forget information, along with a cell state that carries information across long distances. GRUs (Gated Recurrent Units) simplify LSTMs by combining the forget and input gates into a single update gate plus a reset gate, achieving comparable performance with fewer parameters. LSTMs and GRUs power applications in speech recognition, machine translation, and time-series forecasting.

The transformer architecture, based entirely on self-attention, has largely supplanted RNNs for sequence modeling. Self-attention computes a weighted sum over all tokens in a sequence using \( \text{Attention}(Q, K, V) = \text{softmax}(Q K^\top / \sqrt{d}) V \), where weights are derived from learnable query, key, and value projections. Multi-head attention runs several self-attention operations in parallel, each focusing on different representation subspaces, and concatenates their outputs. Because self-attention is permutation-invariant, positional encodings inject token order information. The encoder-decoder architecture maps an input sequence to an output sequence via an encoder that compresses the input and a decoder that generates tokens conditioned on that context. BERT is a bidirectional encoder pre-trained with masked language modeling and next sentence prediction, while GPT is a decoder-only transformer pre-trained with autoregressive next-token prediction; successive GPT versions have scaled to hundreds of billions of parameters and form the basis for ChatGPT-style assistants. Large generative pre-trained models exhibit in-context learning, performing new tasks from examples given in the prompt without any weight updates, and prompt engineering—including few-shot examples, chain-of-thought reasoning, and role assignment—has become a key skill for working with these models.

Natural language processing builds on several foundational techniques. Tokenization splits raw text into smaller units called tokens—words, subwords, or characters. Modern systems typically use subword schemes such as Byte-Pair Encoding (BPE), WordPiece, or SentencePiece, which balance vocabulary size with the ability to handle rare words and morphology. Stemming crudely chops word endings to a rough root (sometimes producing non-words), while lemmatization uses vocabulary and morphology to return proper dictionary forms and is slower but more accurate. Word embeddings are dense, low-dimensional vector representations of words where semantically similar words are nearby. Word2Vec (CBOW and Skip-gram) and GloVe are classic methods; Word2Vec famously produces analogies like king − man + woman ≈ queen, while GloVe factorizes global word co-occurrence matrices. Modern transformer-based models learn contextual embeddings that depend on surrounding text.

Higher-level NLP tasks layer on these foundations. Part-of-speech (POS) tagging assigns a grammatical category to each token, reaching 97%+ accuracy on English with neural sequence models. Named Entity Recognition (NER) locates and classifies entities such as people, organizations, locations, and dates; modern systems use BiLSTM-CRF or transformer-based models. Named entity disambiguation links a mention to a specific entity in a knowledge base—for example, distinguishing "Apple" the company from "apple" the fruit. Dependency parsing analyzes grammatical structure by linking words into typed head–dependent relations. Sentiment analysis classifies the polarity of text (positive, negative, neutral) using lexicon methods, classical ML on TF-IDF features, or fine-tuned transformers, and is widely used for brand monitoring and review analysis.

Contrastive learning has emerged as a powerful paradigm for representation learning. The idea is to pull similar pairs—such as an image and its caption, or two augmentations of the same image—together in embedding space while pushing dissimilar pairs apart. Losses such as InfoNCE, triplet loss, and supervised contrastive loss train models to produce embeddings that cluster semantically related items. CLIP trains an image encoder and a text encoder jointly with a contrastive InfoNCE loss on a massive image-text dataset, aligning the two modalities in a shared space and enabling zero-shot image classification from natural-language prompts. CLIP powers text-to-image systems like DALL·E.

Generative models learn to produce new samples resembling the training distribution. Generative Adversarial Networks (GANs) train two networks in opposition: a generator that produces fake samples from noise and a discriminator that distinguishes real from fake; through adversarial training, the generator produces increasingly realistic outputs and powers deepfakes and high-resolution image synthesis such as StyleGAN. Variational Autoencoders (VAEs) learn a probabilistic latent space by training an encoder to map inputs to a distribution and a decoder to reconstruct from sampled latent codes, optimized via the evidence lower bound; VAEs produce blurry but diverse samples. Diffusion models learn to reverse a gradual noising process—during training, real images are progressively corrupted with noise and the model learns to denoise them, while at inference, samples are generated by denoising from pure noise. Diffusion models such as Stable Diffusion and DALL·E 2 currently lead in image synthesis quality. The Boltzmann machine and its restricted variant (RBM) are older energy-based generative models foundational to deep belief nets. For evaluating text generation, BLEU measures n-gram precision between generated and reference text and is standard for machine translation, while ROUGE measures n-gram recall and is standard for summarization; both have known weaknesses for paraphrases. Discriminative models like logistic regression and SVMs learn P(y|x), while generative models like Naive Bayes, GANs, VAEs, and language models learn P(x, y) or P(x|y) and can sample new data.

Deploying machine learning at scale requires careful systems engineering. Retrieval-Augmented Generation (RAG) augments a language model by retrieving relevant external documents and including them in the prompt. RAG grounds responses in up-to-date, verifiable sources, reducing hallucination, and is cheaper and more dynamic than fine-tuning. A typical RAG pipeline chunks documents, embeds them with a model like a sentence-transformer, indexes the embeddings in a vector database, retrieves the most relevant chunks for a query, and feeds them to the generator. Vector databases such as FAISS, Pinecone, Weaviate, Milvus, and Qdrant store and index high-dimensional embeddings for fast similarity search. At the heart of vector search is approximate nearest neighbor (ANN) retrieval, which trades exactness for massive speedups; algorithms like HNSW (Hierarchical Navigable Small World), IVF, and LSH support billion-scale retrieval in milliseconds. Cosine similarity \( \cos(A, B) = (A \cdot B) / (\|A\| \|B\|) \) is widely used to compare embeddings because it measures directional alignment independent of magnitude. Embedding fine-tuning adjusts pre-trained embeddings on domain-specific data with contrastive losses, often improving retrieval quality where off-the-shelf embeddings miss nuance.

Fine-tuning specializes a pre-trained model for a task. Full fine-tuning updates all weights, which is expensive for large models. LoRA (Low-Rank Adaptation) freezes the original weights and injects trainable low-rank matrices into transformer layers, reducing trainable parameters by 10× to 10000× while matching full fine-tuning quality; multiple LoRA adapters can be swapped for different tasks. Knowledge distillation trains a small student model to mimic a larger teacher, using the teacher's soft probability outputs as additional supervision. Quantization reduces numerical precision (e.g., float32 to int8), shrinking models and speeding inference with minimal accuracy loss; quantization-aware training simulates low precision during training for better results. Pruning removes unimportant weights or entire filters, often combined with fine-tuning to recover accuracy. ONNX provides an open format for representing models, enabling interoperability across frameworks and runtimes like ONNX Runtime and TensorRT.

Recommendation systems predict user preferences and suggest items. Collaborative filtering relies on patterns in user–item interactions without needing item features, using user-based similarity, item-based similarity, or matrix factorization. Matrix factorization decomposes the interaction matrix into low-rank user and item matrices whose dot products approximate observed ratings \[ R \approx U V^\top \] a technique popularized by the Netflix Prize that captures latent preferences and enables efficient recommendations. Content-based methods use item features, and hybrid systems combine both. Modern deep approaches like two-tower neural networks underlie production recommenders. A major challenge is the cold-start problem for new users or items with no interaction history.

Time series forecasting predicts future values from past observations. Classical methods include ARIMA, which combines autoregression, differencing, and moving average components, and Exponential Smoothing. Modern approaches use RNNs, LSTMs, Temporal Convolutional Networks, and transformer-based models such as Informer and PatchTST. Meta's Prophet library is designed for business time series with daily seasonality, holidays, and trend changes, and is robust to missing data and outliers. Stationarity—constant statistical properties over time—is an assumption of many models, while seasonality is a regular periodic pattern to model or remove. The autocorrelation function (ACF) measures correlation between a series and its lagged version, helping identify seasonality and inform model orders. Anomaly detection identifies points that significantly deviate from the rest of the distribution; Isolation Forest isolates anomalies quickly via random splits in trees, and One-class SVM learns a boundary around normal data using a kernel such as RBF. Operationalizing ML introduces challenges like data drift and concept drift, where input distributions or feature–target relationships change over time, requiring monitoring and retraining. MLflow manages the ML lifecycle, including experiment tracking, model packaging, and registries, while feature stores like Feast ensure consistency between offline and online feature pipelines. A/B testing compares model variants by randomly assigning users and measuring business metrics—the gold standard for validating that a new model actually improves outcomes in production.