Data Science Essentials

This deck introduces the core vocabulary and concepts that form the foundation of data science. It covers the basics of what data science is, the typical workflow that practitioners follow, and well-known methodologies like CRISP-DM. You'll also find cards on key building blocks such as datasets, features, and target variables, along with an overview of the main branches of machine learning — including supervised, unsupervised, reinforcement, and semi-supervised learning. Finally, it walks through common task types like regression, classification, clustering, dimensionality reduction, and feature engineering, giving you a well-rounded starting point for the field.

It's a great fit if you're new to data science, preparing for a course or interview, or simply want a refresher on the terminology used in textbooks, courses, and conversations with data teams. The questions are phrased as straightforward definitions, so even if you have no technical background yet, you'll be able to build a solid mental map of how the pieces fit together before diving into code or more advanced topics.

Because many of these terms are closely related — for example, regression and classification both fall under supervised learning — it helps to study the deck across several short sessions rather than cramming everything at once. Try reviewing a small batch of cards each day, and when you come across a concept you've already seen, take a moment to connect it back to the bigger picture. This kind of spaced repetition will make the vocabulary feel intuitive and easier to recall when you encounter it in articles, courses, or on the job.

Data science is an interdisciplinary field that combines statistics, programming, and domain expertise to extract meaningful insights from data. A data scientist typically draws on tools from mathematics, computer science, and the application area in question, using code to turn raw observations into decisions, predictions, or scientific findings. The work is rarely a single step; rather, it follows a well-known lifecycle that begins with defining the problem, moves through data collection, cleaning, exploration, modeling, and evaluation, and ends with deploying the result so it can be used by stakeholders. This lifecycle is intentionally iterative: findings in evaluation often send the practitioner back to cleaning or modeling.

To make this lifecycle repeatable, teams often adopt a structured methodology such as CRISP-DM, the Cross-Industry Standard Process for Data Mining. CRISP-DM formalizes the stages above and emphasizes that understanding the business problem and the data come before any algorithm is chosen. At the heart of any project sits the dataset: a collection of related data usually arranged as rows (observations) and columns (variables). Each column used as input is called a feature, a measurable property such as age, price, or pixel intensity, while the column the model tries to predict is called the target variable. Distinguishing features from the target is the first move in translating a real-world question into a machine learning problem.

Different problems call for different kinds of learning. Supervised learning trains on labeled examples to predict outcomes; unsupervised learning seeks patterns such as groups or low-dimensional summaries in unlabeled data; reinforcement learning has an agent learn by taking actions and receiving rewards or penalties; and semi-supervised learning blends a small amount of labeled data with a larger pool of unlabeled data. Within supervised learning the target's data type determines the task: regression predicts a continuous numeric output, classification predicts a categorical label, and clustering (an unsupervised counterpart) groups similar points without predefined labels. Dimensionality reduction, meanwhile, compresses many features into fewer while preserving information, often as a preprocessing step or for visualization.

The people doing this work have distinct roles. A data scientist focuses on analysis and modeling, framing questions and choosing methods. A data engineer builds and maintains the pipelines, warehouses, and infrastructure that move and shape data at scale. A data analyst typically sits closer to reporting and business intelligence, querying data with SQL and building dashboards. These roles overlap, but recognizing them helps clarify who owns which part of the pipeline, from raw ingestion to communicated insight.

Real datasets arrive messy, and most of a practitioner's time goes into preparing them. Data cleaning is the process of detecting and correcting errors such as missing values, duplicates, and outliers. Missing data — values recorded as NaN or NULL — can be addressed by dropping affected rows or columns, by imputation using the mean, median, or mode, by more sophisticated model-based imputation, or by choosing algorithms that are robust to missingness. Outliers are points that differ sharply from the rest of the data, and they can be flagged through z-scores, the interquartile range (IQR), boxplots, or model-based methods such as isolation forests. The choice depends on whether an outlier represents an error to remove or a genuine extreme that the model should learn from.

Before modeling, exploratory data analysis (EDA) is the initial investigation that uncovers distributions, anomalies, and early hypotheses. EDA guides what cleaning and engineering steps are actually needed. Two common numerical preparations are normalization, which scales features to a common range such as \([0,1]\), and standardization, which centers features to mean 0 and standard deviation 1. These steps matter because many algorithms are sensitive to the scale of their inputs, and distance-based or gradient-based methods in particular behave poorly when one feature dominates others by virtue of its units.

Categorical variables require their own transformations. One-hot encoding converts a category into a set of binary columns, one per level; label encoding maps each category to an integer; and target encoding replaces a category with the mean of the target variable within that category, which can capture predictive signal but risks data leakage if applied carelessly. Beyond encoding, feature engineering creates new features from raw data — ratios, time deltas, interaction terms, text statistics — to better expose the underlying signal. Feature selection, by contrast, prunes the set of features down to the most relevant ones, reducing noise, speeding training, and sometimes improving generalization. In very high-dimensional settings, practitioners face the curse of dimensionality: distances become uninformative and data becomes sparse, motivating dimensionality reduction methods discussed in a later chapter.

Class imbalance, where one class has far more examples than another, can cripple a naive classifier. It is typically handled by resampling — oversampling the minority, undersampling the majority — or by using synthetic methods like SMOTE, which creates new minority examples by interpolating between existing ones. Class weighting and reframing the problem as anomaly detection are alternative strategies. Together, these preparation choices often determine whether a model is useful at all.

To estimate how a model will perform on unseen data, practitioners split the data into training and testing sets. This simple train-test split protects against overfitting, the failure mode in which a model performs well on training data but poorly on new data. A more reliable estimate comes from cross-validation, where the data is divided into folds and each fold is used in turn as the test set while the others train the model. K-fold cross-validation with \(k\) folds averages the result across all \(k\) test runs. When class proportions are important, stratified k-fold preserves those proportions in every fold, giving more stable estimates under imbalance.

Closely related to overfitting is data leakage, the contamination of training by information that would not be available at prediction time — for example, imputing missing values using statistics computed on the full dataset, or peeking at test labels during feature engineering. Leakage inflates performance estimates and produces models that collapse in production. Underfitting is the opposite failure: the model is too simple to capture the patterns in the data. The bias-variance tradeoff describes the tension between underfitting (high bias, low variance) and overfitting (low bias, high variance), and most modeling choices are implicitly attempts to balance the two. Regularization is one such mechanism: it adds a penalty for complexity to the loss function. L1 regularization (Lasso) penalizes the absolute values of coefficients and pushes some to zero, producing sparse models, while L2 regularization (Ridge) penalizes squared coefficients and shrinks them smoothly. Elastic Net combines L1 and L2.

Training a model usually means minimizing a loss function that quantifies how wrong predictions are. For regression, common losses include mean squared error (MSE), the average of squared differences between predictions and actuals, and mean absolute error (MAE), the average of absolute differences. For classification, cross-entropy loss measures the distance between predicted probabilities and the true distribution. The optimizer that drives this minimization is typically gradient descent, which iteratively updates parameters in the direction opposite the gradient of the loss. The learning rate controls the step size of each update; too large and the model diverges, too small and training crawls. Stochastic gradient descent uses single examples or small batches (mini-batch gradient descent) to make updates cheap and noisy enough to escape poor local minima, at the cost of a less stable trajectory.

Beyond the loss, classification models are evaluated with metrics tailored to the problem. Accuracy is the fraction of correct predictions, but it can mislead on imbalanced data; precision (\( \text{TP} / (\text{TP}+\text{FP}) \)) measures how many of the predicted positives are real, while recall (\( \text{TP} / (\text{TP}+\text{FN}) \)) measures how many of the actual positives were caught. The F1 score is the harmonic mean of precision and recall, useful when both matter. The confusion matrix lays out predicted versus actual counts for each class, supporting derived metrics. Threshold-based summaries visualize the trade-off: the ROC curve plots true positive rate against false positive rate at various thresholds and the AUC summarizes discrimination as a single number; the precision-recall curve is more informative when the positive class is rare. Statistical significance, hypothesis testing, and p-values complement these metrics by helping judge whether observed differences are likely due to chance under a null hypothesis.

Linear regression models a continuous target as a linear combination of features, predicting \( \hat{y} = w^\top x + b \). It is interpretable, fast, and a baseline against which more elaborate models are measured. Logistic regression adapts the same linear idea to classification by passing the linear score through the logistic (sigmoid) function to produce a class probability, then training via maximum likelihood. Despite their names, both are linear models, and both are surprisingly strong baselines on many problems.

Decision trees take a different tack: they recursively split the data on feature values, creating a tree of if-then rules. Each split is chosen to reduce impurity, measured by entropy (the reduction is called information gain) or by Gini impurity in the CART family. Trees are highly interpretable and handle mixed data types, but a single tree tends to overfit. Ensemble methods fix this. A random forest trains many decision trees on bootstrap samples of the data and averages their predictions — a strategy called bagging, short for bootstrap aggregating. Boosting takes the opposite approach: it trains models sequentially, with each new model focusing on the errors of the previous ones. Gradient boosting fits each new model to the negative gradient (residuals) of the loss, and efficient implementations such as XGBoost, LightGBM, and CatBoost (which handles categorical features natively) have made boosting the dominant approach for tabular data.

Several other classical algorithms remain widely used. K-nearest neighbors (KNN) is a non-parametric method that classifies a new point by the majority label of its closest training examples, with the distance metric doing much of the work. Naive Bayes is a probabilistic classifier that applies Bayes' theorem under a strong, often violated assumption of feature independence; remarkably, it still performs well in text classification and other high-dimensional problems. Support vector machines (SVMs) find the hyperplane that maximally separates classes, and the kernel trick implicitly maps data into higher-dimensional spaces where a linear separator may exist, enabling nonlinear boundaries without computing the mapping explicitly.

Choosing among these algorithms depends on the data size, dimensionality, interpretability requirements, and whether the task is regression or classification. In practice, gradient-boosted trees, logistic regression, and random forests are common starting points on tabular data, while KNN, naive Bayes, and SVMs retain niches in text, small-data, or geometrically structured problems.

When labels are absent, unsupervised methods extract structure directly from the data. Clustering algorithms group similar points: k-means partitions data into \(k\) clusters by minimizing within-cluster variance, hierarchical clustering builds a tree of nested groups by successive merging (agglomerative) or splitting (divisive), and DBSCAN finds clusters as dense regions separated by sparse noise, which lets it discover clusters of arbitrary shape and identify outliers. Choosing \(k\) in k-means is often guided by the elbow method, which plots the within-cluster cost against \(k\) and looks for a bend, and by the silhouette score, which measures how similar each point is to its own cluster compared to neighboring clusters.

Dimensionality reduction compresses many features into fewer while preserving important structure. Principal Component Analysis (PCA) finds orthogonal directions of maximum variance, producing a linear, interpretable embedding. For visualization and structure discovery in high dimensions, nonlinear methods such as t-SNE and UMAP are popular; UMAP in particular is fast, scalable, and tends to preserve more global structure than t-SNE. These techniques are invaluable for both exploration and as preprocessing for downstream models.

Deep learning extends the same learning paradigm with flexible, layered models. A neural network is composed of layers of interconnected nodes, or neurons; the simplest is the perceptron, a single linear classifier. A multilayer perceptron (MLP) stacks fully connected hidden layers between input and output. Each neuron applies an activation function to its weighted sum. ReLU, defined as \(\max(0, x)\), is the default choice for hidden layers due to its well-behaved gradients; sigmoid, \(1/(1+e^{-x})\), squashes outputs to \((0,1)\) and is used for binary probabilities; and softmax converts a vector of logits into a probability distribution over classes. Training these networks is driven by backpropagation, an efficient algorithm that computes gradients of the loss with respect to each parameter by applying the chain rule from the output back through the layers.

Specialized architectures target specific data modalities. Convolutional neural networks (CNNs) use convolutional layers to exploit spatial structure in grid-like data such as images, making them the workhorse of computer vision. Recurrent neural networks (RNNs) handle sequences by passing a hidden state between timesteps; LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) address the difficulty RNNs have capturing long-range dependencies through gating mechanisms. Transformers replace recurrence with self-attention, a mechanism that lets each position in a sequence attend to all others, and they are the foundation of modern NLP. BERT uses a bidirectional transformer encoder for representation learning, while GPT uses an autoregressive transformer decoder for generation. Adapting these large pretrained models is rarely done from scratch: transfer learning reuses a model trained on one task for another related task, and fine-tuning continues its training on task-specific data to specialize it. Training is stabilized and regularized by techniques such as dropout (randomly disabling neurons during training), batch normalization (normalizing layer inputs to accelerate and stabilize training), early stopping (halting when validation performance plateaus), and data augmentation (creating new training examples by transforming existing ones, especially important in vision).

Statistics is the language with which claims are tested and uncertainty is communicated. Exploratory analysis generates hypotheses from the data; confirmatory analysis then tests them with formal procedures. Hypothesis testing evaluates a claim about a population by computing a p-value, the probability of observing data at least as extreme as what was measured, assuming the null hypothesis is true. A result is declared statistically significant when this probability is below a chosen threshold, typically 0.05. Bootstrapping, resampling with replacement, provides an alternative route to the same kind of inference: it builds an empirical sampling distribution from the data itself, from which confidence intervals are read. A confidence interval is a range that, under repeated sampling, contains the true parameter with a stated frequency — a frequentist interpretation that contrasts with the Bayesian view, where probability reflects updated beliefs given priors and observed data via Bayes' theorem.

Correlation quantifies a linear association between variables; causation asserts that one variable influences another. The distinction is critical, because correlation does not imply causation: confounding variables that affect both predictor and outcome can produce spurious relationships. Simpson's paradox dramatizes the point — a trend that holds for the population can reverse when the data is split into groups. Careful study design and adjustment for confounders are the usual defenses.

As models grow more complex, interpretability becomes essential. Model interpretability is the degree to which a human can understand a model's decisions. SHAP (SHapley Additive exPlanations) attributes a prediction to features using game-theoretic Shapley values, giving a consistent, locally accurate explanation. LIME (Local Interpretable Model-agnostic Explanations) approximates the model locally with an interpretable surrogate to explain individual predictions. Permutation importance offers a more global view: shuffle a feature and observe how much performance drops, capturing reliance on that feature. These tools help practitioners debug models, build trust with stakeholders, and satisfy regulatory or ethical review.

Fairness in machine learning asks whether models systematically disadvantage protected groups. Bias in data — historical skews, underrepresentation, label noise that correlates with identity — can flow into models as disparate outcomes. Quantitative criteria formalize the goal: demographic parity asks for similar outcome rates across groups, while equalized odds asks for equal true positive and false positive rates across groups. These criteria can be in tension with each other and with accuracy, so choices must be deliberate and documented. Privacy-preserving methods like differential privacy add calibrated noise so that an individual's data does not significantly affect the output, and federated learning trains models on decentralized devices without centralizing raw data, reducing exposure risk while still producing useful models.

Python is the dominant language in data science, supported by a layered ecosystem. NumPy provides N-dimensional arrays and fast vectorized operations; pandas offers DataFrames for tabular manipulation; scikit-learn implements classical machine learning algorithms with a consistent API; matplotlib is the foundational plotting library, complemented by seaborn for statistical graphics and Plotly for interactive visualizations; and Jupyter Notebooks provide an interactive environment that interleaves code, results, and narrative. R remains popular in academia and finance for statistical modeling, while SQL is the lingua franca for querying and manipulating relational databases, supporting joins to combine rows from related tables, GROUP BY for aggregation, and window functions for calculations across related rows.

Behind these familiar interfaces lies substantial infrastructure. ETL — Extract, Transform, Load — moves and shapes data into analytical systems; ELT flips the order, doing transformation in the warehouse where compute is cheap and the raw data is preserved. Storage architectures include data warehouses, central repositories optimized for analytical reads; data lakes, which store raw structured and unstructured data at scale; and lakehouses, a hybrid exemplified by Delta Lake and Apache Iceberg. Distributed processing is enabled by Apache Hadoop, with HDFS providing fault-tolerant storage and MapReduce offering a parallel programming model; Apache Spark supersedes MapReduce with in-memory computation; Apache Kafka handles real-time streaming; and Apache Airflow orchestrates workflows. Cloud platforms reduce the operational burden: Snowflake separates storage and compute in a managed warehouse, BigQuery offers Google’s serverless alternative, and Databricks unifies analytics on Spark. dbt applies SQL and Jinja templating for in-warehouse transformations, and feature stores centralize curated, versioned features so training and inference see the same definitions. Hadoop's broader ecosystem, including HDFS and MapReduce, remains relevant in legacy and on-premise installations.

Model parameters are learned from data during training, whereas hyperparameters such as the learning rate or tree depth are configuration choices set before training. Tuning them effectively is its own discipline: grid search exhaustively tries combinations, random search samples them, and Bayesian optimization builds a probabilistic model of performance to choose promising points efficiently. Once trained, models enter the MLOps lifecycle: deployment, monitoring, and retraining. Online inference serves predictions in real time, often behind a REST API, while batch inference generates predictions on bulk data at scheduled intervals. Serving systems such as TensorFlow Serving and NVIDIA's Triton Inference Server optimize throughput and latency, and ONNX provides a portable model interchange format so a model trained in one framework can run in another. A reusable feature pipeline computes features identically for training and inference, preventing skew. Data versioning with tools like DVC and lifecycle management with MLflow keep experiments, datasets, and models reproducible.

Models degrade as the world changes. Model drift is a decline in performance over time, driven by data drift (changes in input distributions), concept drift (changes in the relationship between inputs and outputs), or both. Monitoring catches both the symptoms (accuracy drops) and the causes (distribution shifts in features). Data quality monitoring platforms such as Great Expectations continuously validate data against expectations, complementing observability tooling that infers a system’s internal state from its outputs. Impact is measured through A/B testing, which compares two treatments to estimate causal effects, complementing offline metrics with real-world evidence.

Several specialized domains deserve their own methods. Natural Language Processing (NLP) analyzes and generates text. Its pipeline begins with tokenization, splitting text into words, subwords, or characters, often after removing common stop words such as “the” and “is”. Words are reduced to a common form by stemming (a rule-based chop) or lemmatization (a context-aware reduction to a dictionary form). Representations matter: TF-IDF weights words by their importance within a document relative to a corpus, while dense word embeddings such as Word2Vec and GloVe capture semantics in vectors, so that similar words cluster in space. These foundations power modern language models and tasks from sentiment analysis to machine translation. Computer vision, the interpretation of visual information, is built around tasks such as image classification (assigning a label to an image), object detection (locating and classifying objects within an image), and image segmentation (partitioning an image into meaningful regions). Recommendation systems suggest items users may like: collaborative filtering leverages similarity among users or items, while content-based filtering relies on item attributes, with hybrid systems common in practice.

Data quality is the fitness of data for its intended use, measured along dimensions including accuracy, completeness, consistency, timeliness, validity, and uniqueness. Data governance encompasses the policies and processes for managing data assets — who can access what, how data is defined, and how it is audited. Supporting practices include data catalogs (metadata inventories of available assets), data lineage (tracking the origin and transformations of data), and data dictionaries that define fields and their meanings. Database schemas — the structure of columns, types, and constraints — underpin these activities. Database normalization reduces redundancy by organizing tables into well-separated forms, while denormalization trades redundancy for query speed, a useful choice in analytical stores where read performance dominates.

Analytical and transactional workloads have different shapes. OLTP (Online Transaction Processing) handles high-volume, short transactions, while OLAP (Online Analytical Processing) serves multi-dimensional analytical queries. Data warehouses often use a star schema, with a central fact table of measurable events surrounded by dimension tables that provide descriptive context; the snowflake schema is a more normalized variant. Slowly changing dimensions (SCDs) handle changes to dimensional data, with Type 2 SCD preserving history by adding new rows. To keep load times manageable as data grows, practitioners use incremental loading (loading only new or changed data), partitioning (dividing data into smaller pieces for performance), and sharding (distributing data across multiple databases). Primary keys uniquely identify each row, foreign keys link rows across tables, and OLTP/OLAP distinctions shape how these are chosen and indexed.

Results only matter if they are understood and acted upon. Dashboards built with Tableau, Power BI, or Looker surface key metrics for ongoing monitoring, while tools like Streamlit and Gradio let practitioners quickly build interactive data apps and model demos. Platforms such as Kaggle provide shared datasets and competitions that sharpen skills and benchmark methods. Storytelling with data — crafting a clear narrative around findings — turns numbers into decisions, which is ultimately the purpose of the entire pipeline. Ethical AI extends this responsibility by demanding that systems align with human values and avoid harm. Goodhart's Law captures a recurring trap: "When a measure becomes a target, it ceases to be a good measure." Optimization against a single metric in data science, as elsewhere, can distort the underlying goal, so the questions a model is asked to answer deserve the same care as the answers themselves.