Statistics Essentials

This deck walks you through the foundational building blocks of statistics, starting with the difference between a population and a sample and why researchers usually work with samples rather than collecting data from everyone. You'll then move into the distinction between parameters and statistics, followed by an introduction to descriptive and inferential statistics — two branches that answer different kinds of questions about data. The second half of the deck focuses on the most common ways to summarize a dataset, including measures of center like the mean, median, and mode, as well as measures of spread such as the range, variance, and standard deviation.

It's a great fit if you're just starting out in a statistics or data science course, preparing for an exam, or returning to the subject after some time away and want a clean refresher on the vocabulary and core ideas. Even if you've seen these terms before, the flashcard format encourages you to recall definitions from memory, which is a much stronger test of understanding than simply recognizing them on a page.

Because most of these cards are definition-based, spaced repetition works especially well — short, frequent review sessions will help the terminology stick far better than one long cram. As you study, try connecting each new term to a small mental example, like a class of students or a week's worth of test scores, so the formulas feel like tools for solving real problems rather than abstract rules to memorize.

Statistics is the science of learning from data, and every statistical study begins by clarifying what is being measured and from whom. The entire collection of individuals, items, or measurements that share a common property is called the population, while a subset selected from that population for analysis is called a sample. Because measuring every member of a population is often impractical, costly, or even impossible, statisticians rely on samples to draw conclusions about larger groups. This practical necessity gives rise to a crucial distinction: a parameter is a numerical characteristic of a population (such as the population mean \(\mu\) or standard deviation \(\sigma\)), whereas a statistic is a numerical characteristic computed from a sample (such as the sample mean \(\bar{x}\) or sample standard deviation \(s\)).

Statistics is commonly divided into two broad branches. Descriptive statistics encompasses the tools used to summarize and present the main features of a dataset — its center, spread, and shape — without attempting to generalize beyond the data at hand. Inferential statistics, by contrast, provides the methods for moving from sample data to claims about a wider population, using probability theory to quantify uncertainty. Understanding this distinction is the foundation for everything that follows: descriptive tools help us see what our data look like, while inferential tools tell us what those data allow us to say about the world.

The most common numerical summaries describe a dataset's center. The mean is the arithmetic average — the sum of all values divided by the count — while the median is the middle value of an ordered dataset (or the average of the two middle values when the count is even). The mode is the value or values that occur most frequently. When data are skewed or contain extreme outliers, the median is preferred because it is robust: a single unusually large or small value can pull the mean dramatically but barely affects the median. To describe spread, statisticians use the range (maximum minus minimum), variance (the average of squared deviations from the mean), and standard deviation (the square root of the variance, which is expressed in the original units of the data). The interquartile range, defined as \(\text{IQR} = Q_3 - Q_1\), captures the spread of the middle 50% of the data, where the quartiles divide ordered data into four equal parts. Percentiles generalize this idea: the k-th percentile is the value below which k% of the observations fall. The z-score, computed as \(z = (x - \mu) / \sigma\), expresses how many standard deviations a value lies from the mean and is a useful way to standardize observations across different scales.

The shape of a distribution also matters. A positively skewed distribution has a long right tail and a mean greater than the median, while a negatively skewed distribution has a long left tail and a mean less than the median. Kurtosis measures the "tailedness" or peakedness of a distribution relative to the normal. These numerical summaries are paired with visual displays: histograms show the frequency distribution of a continuous variable with adjacent bars, boxplots summarize a dataset using its minimum, quartiles, and maximum while flagging outliers, and scatterplots display pairs of numerical values to reveal relationships between two variables. Choosing the right summary and the right plot depends on the type of variable at hand: categorical variables take values from named categories, numerical variables take numeric values, and numerical variables are further split into discrete (countable, such as the number of children) and continuous (any value in an interval, such as weight).

Probability provides the mathematical language for uncertainty. A probability is a number between 0 and 1 that quantifies how likely an event is to occur. Several rules govern how probabilities combine. The complement rule states that \(P(\text{not } A) = 1 - P(A)\). The addition rule gives \(P(A \text{ or } B) = P(A) + P(B) - P(A \text{ and } B)\), and the multiplication rule states \(P(A \text{ and } B) = P(A) \cdot P(B \mid A)\). Two events are independent if \(P(A \text{ and } B) = P(A) \cdot P(B)\), meaning that knowing one tells you nothing about the other. They are mutually exclusive if \(P(A \text{ and } B) = 0\), because they cannot occur together. Conditional probability, \(P(A \mid B) = P(A \text{ and } B) / P(B)\), expresses the probability of A given that B has occurred.

Bayes' theorem, \(P(A \mid B) = P(B \mid A) \cdot P(A) / P(B)\), formalizes how evidence updates beliefs. In Bayesian language, the prior \(P(A)\) is the probability of a hypothesis before seeing the data, the likelihood \(P(\text{data} \mid \text{hypothesis})\) measures how plausible the data are under that hypothesis, and the posterior \(P(A \mid B)\) is the updated belief after combining the prior with the observed evidence. The law of total probability, \(P(A) = \sum_i P(A \mid B_i) \cdot P(B_i)\) over a partition \(\{B_i\}\), ties conditional and unconditional probabilities together. A random variable is a numerical outcome of a random phenomenon; it is discrete when it takes countable values and continuous when it takes any value in an interval. For discrete variables, the probability mass function (PMF) gives \(P(X = x)\) for each possible value, while for continuous variables the probability density function (PDF) has the property that the area under its curve over an interval equals the probability of falling in that interval. The cumulative distribution function \(F(x) = P(X \le x)\) describes either case, and the expected value \(E[X] = \sum x \cdot P(X = x)\) for discrete variables represents the long-run average. The variance \(E[(X - E[X])^2]\) is the expected squared deviation from this mean. When arranging outcomes, permutations \(_nP_r = n!/(n-r)!\) count ordered selections, while combinations \(_nC_r = n!/(r!(n-r)!)\) count unordered selections.

Several named probability distributions serve as building blocks for statistical models. The Bernoulli distribution models a single trial with two outcomes, success with probability \(p\) and failure with probability \(1 - p\). Repeating such trials independently yields the binomial distribution, which counts the number of successes in \(n\) Bernoulli trials; it has mean \(np\) and variance \(np(1-p)\). The Poisson distribution models the count of events occurring in a fixed interval at a constant rate \(\lambda\), with both its mean and variance equal to \(\lambda\). The geometric distribution counts the number of trials needed to obtain the first success in a sequence of independent Bernoulli trials.

For continuous variables, the normal distribution is the most important: a symmetric, bell-shaped curve fully described by its mean \(\mu\) and standard deviation \(\sigma\). The standard normal is the special case with mean 0 and standard deviation 1, and the 68-95-99.7 rule states that about 68%, 95%, and 99.7% of observations fall within 1, 2, and 3 standard deviations of the mean, respectively. The t-distribution resembles the normal but has heavier tails, making it appropriate when the sample size is small or the population standard deviation is unknown. The chi-square distribution is the distribution of a sum of squared standard normal variables and is used in tests involving variance or categorical data, while the F-distribution, the ratio of two scaled chi-square variables, underpins ANOVA and many regression diagnostics. Two theoretical results unify much of this material. The central limit theorem states that the sampling distribution of the mean approaches a normal distribution as sample size grows, regardless of the population's underlying shape; this is what justifies normal-based inference for many real-world data. The law of large numbers guarantees that the sample mean converges to the population mean as the sample grows, so averages become more reliable with more data.

Inferential statistics depends on how data are collected. In simple random sampling, every member of the population has an equal chance of selection; in stratified sampling, the population is divided into strata and a random sample is drawn from each, which can improve precision when strata differ. Cluster sampling selects whole groups rather than individuals and reduces cost, while systematic sampling selects every k-th element after a random start. Biases threaten validity at every stage: selection bias arises from how units are included, response bias from inaccurate answers, non-response bias from systematic differences between respondents and non-respondents, and sampling bias generally from any non-random mechanism that distorts estimates.

A sampling distribution is the probability distribution of a statistic across many samples of the same size, and its standard deviation is the standard error. For the sample mean, the standard error is \(\sigma/\sqrt{n}\). A point estimate is a single value used to approximate a parameter, while an interval estimate such as a confidence interval gives a range of plausible values. A 95% confidence interval should be interpreted carefully: if sampling were repeated many times, about 95% of such intervals would contain the true parameter. Higher confidence levels produce wider intervals because more certainty requires more room. The sample mean is an unbiased estimator of the population mean, meaning its expected value equals \(\mu\); dividing the sum of squared deviations by \(n-1\) rather than \(n\) in the sample variance — Bessel's correction — likewise makes the sample variance an unbiased estimator of \(\sigma^2\). The maximum likelihood estimator (MLE) is the parameter value that maximizes the probability of the observed data.

Hypothesis testing is the formal procedure for evaluating claims about a population. The null hypothesis \(H_0\) is the default assumption, usually of no effect or no difference; the alternative \(H_1\) is the claim being tested. The p-value is the probability, assuming \(H_0\) is true, of observing data at least as extreme as those obtained. The significance level \(\alpha\) (often 0.05) is a pre-chosen threshold: if the p-value falls below it, the null hypothesis is rejected in favor of the alternative. Two kinds of errors are possible: a Type I error rejects a true null (a false positive), while a Type II error fails to reject a false null (a false negative). Statistical power, \(1 - \beta\), is the probability of correctly rejecting a false null, and it increases with larger samples, larger effects, lower variance, and a higher \(\alpha\). Tests may be one-tailed (a directional alternative such as \(\mu > 5\)) or two-tailed (any difference such as \(\mu \ne 5\)). Even when an effect is statistically significant, it may lack practical significance — meaning it is not large enough to matter in the real world, regardless of p-value.

The choice of test depends on the data and the question. A z-test is appropriate when the population standard deviation is known or the sample is large, whereas a t-test is used when the population standard deviation is unknown and must be estimated from the sample. An independent samples t-test compares the means of two unrelated groups, while a paired t-test compares two related measurements on the same subjects (such as before-and-after observations). Degrees of freedom, often \(n - 1\) for variance estimates, describe the number of values free to vary after constraints are imposed.

When comparing three or more groups, ANOVA (analysis of variance) tests whether any group means differ, using an F-statistic that contrasts between-group variance with within-group variance. A significant ANOVA indicates that at least one mean differs, but it does not say which; post-hoc tests such as Tukey's HSD perform the necessary pairwise comparisons. For categorical data, the chi-square test of independence assesses whether two categorical variables are associated, while the chi-square goodness-of-fit test checks whether observed frequencies match expected proportions. When distributional assumptions like normality are doubtful, non-parametric tests offer robust alternatives: the Mann-Whitney U test compares two independent samples, the Wilcoxon signed-rank test handles paired samples using ranks, and the Kruskal-Wallis test generalizes to several groups, paralleling one-way ANOVA.

Correlation quantifies the strength and direction of a linear relationship between two numerical variables. Pearson's correlation coefficient \(r\) ranges from \(-1\) to \(+1\); values near \(0\) indicate no linear association, although a non-linear relationship may still exist. Spearman's correlation captures monotonic (rather than strictly linear) association by working with ranks. Importantly, correlation does not imply causation — apparent associations can arise from coincidence, confounding variables that influence both predictors and outcomes, or reverse causation. Simple linear regression models a straight-line relationship \(\hat{y} = b_0 + b_1 x\) between a single independent variable and a dependent variable. The least squares method chooses the coefficients that minimize the sum of squared residuals, where a residual is the difference \(y - \hat{y}\) between an observed and predicted value. R-squared measures the proportion of variation in the dependent variable explained by the model; for example, \(R^2 = 0.8\) means that 80% of the variability in the response is captured.

Multiple regression extends this idea to two or more independent variables, where each regression coefficient is interpreted as the expected change in \(y\) for a one-unit increase in that predictor, holding other variables constant. Categorical predictors enter the model as dummy variables, with one level designated as the reference category against which the others are compared. Interaction terms, written as products \(x_1 \cdot x_2\), capture how one predictor's effect depends on the level of another. Logistic regression handles a binary outcome by passing a linear combination of predictors through the logistic function to produce a probability; the logit link \(\log(p/(1-p))\) linearizes this relationship, and exponentiating a coefficient yields an odds ratio, the multiplicative change in odds for a one-unit increase in the predictor. Several diagnostic issues can undermine a regression: multicollinearity arises when independent variables are highly correlated and destabilizes coefficient estimates; heteroscedasticity, often visible as a funnel shape in a residual plot of residuals versus fitted values, signals non-constant error variance; outliers, leverage points (extreme x-values), and influential points (those whose removal substantially changes the fit) are flagged using measures such as Cook's distance.

Choosing among models involves the bias-variance tradeoff: simple models underfit and miss structure, while complex models overfit and capture noise, leading to poor generalization. Cross-validation estimates true predictive performance by splitting data into training and validation folds; in k-fold cross-validation this is repeated for k rotations. Regularization discourages overfitting by penalizing complexity: L1 (Lasso) adds a penalty proportional to \(|\beta|\) and can shrink coefficients exactly to zero, performing feature selection, while L2 (Ridge) adds a penalty proportional to \(\beta^2\) and shrinks coefficients toward zero without eliminating them. Information criteria provide further tools for model selection: AIC penalizes complexity gently via \(2k - 2\ln(L)\), while BIC's \(k \ln(n) - 2\ln(L)\) penalizes it more heavily. Other regression diagnostics include Q-Q plots, which compare sample quantiles to those of a reference distribution to check normality, and the Shapiro-Wilk test, which formally tests the null hypothesis of normality.

Beyond the core toolkit, several specialized areas round out statistical practice. A time series is a sequence of observations recorded over time; such data exhibit autocorrelation (correlation with a lagged version of themselves) and may or may not be stationary (having constant mean and variance over time). A moving average smooths a series by averaging values within a sliding window, and ARIMA models combine autoregressive and moving-average components with differencing to capture a wide range of temporal patterns. Experimental design emphasizes the difference between observational studies, in which researchers merely record what they see, and experimental studies, in which they actively manipulate variables. Random assignment of subjects to conditions balances unmeasured confounders across groups and is what enables causal inference; placebos and double-blind procedures, in which neither participants nor researchers know who receives which treatment, control for psychological effects. Control groups provide a baseline against which treatment groups are compared, and designs may be within-subjects (every subject experiences all conditions) or between-subjects (different subjects receive different conditions).

Probability itself admits two interpretations. The frequentist view treats probability as the long-run frequency of an event in repeated trials, with confidence intervals interpreted as long-run coverage rates. The Bayesian view treats probability as a degree of belief, updated through evidence; here, a credible interval directly expresses the probability that the parameter lies in a given range, and conjugate priors — priors chosen so that the posterior belongs to the same family — make computation especially clean. Two resampling-based methods complement parametric inference: bootstrapping estimates sampling distributions by drawing samples with replacement from the observed data, while permutation tests build a null distribution by shuffling labels. When synthesizing evidence across many studies, meta-analysis combines effect estimates and their confidence intervals (often displayed in a forest plot) and uses funnel plots to detect publication bias.

A surprising number of pitfalls arise from human judgment rather than mathematics. Regression to the mean describes the tendency for extreme observations to be closer to the mean on subsequent measurements. Simpson's paradox occurs when a trend visible in aggregated data reverses once subgroups are considered. The gambler's fallacy and the hot-hand fallacy both reflect mistaken beliefs that past random outcomes influence future independent trials. The base rate fallacy arises when people ignore background frequencies in favor of case-specific information, and survivorship bias results from focusing only on entities that "survived" a selection process. In research, publication bias and p-hacking (manipulating analyses until results become significant) distort the published literature, while HARKing — Hypothesizing After Results are Known — presents post-hoc findings as if they were planned. Multiple testing inflates the Type I error rate, but corrections such as the Bonferroni adjustment (dividing \(\alpha\) by the number of tests) and false discovery rate methods like Benjamini-Hochberg control this problem. Finally, statistical learning blends statistics with machine learning, distinguishing supervised learning (fitting models from labeled input-output pairs) from unsupervised learning (finding structure such as clusters in unlabeled data). Techniques such as principal component analysis reduce dimensionality by identifying orthogonal directions of maximum variance, while k-means clustering partitions data into k groups by minimizing within-cluster variance, providing a glimpse into the broader world of data-driven discovery that statistics now supports.