Statistics And Probability

This deck walks you through the foundational concepts of statistics and probability, making it a great starting point if you're new to these subjects or looking to refresh the basics. You'll find cards covering essential descriptive statistics like mean, median, mode, variance, and standard deviation, along with tools for understanding data spread such as quartiles, the interquartile range, and outlier detection. The probability side introduces you to key ideas including the addition and multiplication rules, mutually exclusive and independent events, and conditional probability.

It's well suited for students taking an introductory statistics or data analysis course, learners preparing for exams, or anyone wanting to build a solid vocabulary for working with data. The questions are framed in a straightforward Q&A style, so you can test yourself on both definitions and the reasoning behind why certain measures are used in different situations.

Because many of these concepts build on one another, it's helpful to study them in small chunks rather than all at once. Try reviewing a few cards per session and coming back to them over several days so the formulas and distinctions, like the difference between population and sample standard deviation, really stick. Working through a couple of practice problems on paper after each study session can also reinforce how these ideas apply in practice.

Statistics begins with the task of summarizing data, and the first step is choosing meaningful measures of central tendency. The mean, calculated as \(\bar{x} = \sum x_i / n\), captures the arithmetic average and is the most widely used summary, but it is sensitive to extreme values. The median, the middle value of sorted data (or the average of the two middle values when the count is even), is more robust to outliers and is therefore preferred when distributions are skewed. The mode identifies the most frequently occurring value, and a dataset may be unimodal, bimodal, or multimodal, or have no mode at all if no value repeats.

Beyond central tendency, we measure how spread out the data is. The range, equal to the maximum minus the minimum, is the simplest measure of dispersion, but it captures only the extremes and ignores how data is distributed between them. The variance \(\sigma^2 = \sum (x_i - \bar{x})^2 / n\) and its square root, the standard deviation \(\sigma\), capture the average deviation from the mean and are the most informative measures of spread. Standard deviation is expressed in the original units of the data, making it easier to interpret than variance, which is in squared units. When estimating population variance from a sample, dividing by \(n - 1\) instead of \(n\), a correction known as Bessel's correction, yields an unbiased estimate.

Quartiles offer a more nuanced view of spread by dividing sorted data into four equal parts: Q1 at the 25th percentile, Q2 at the median, and Q3 at the 75th percentile. The interquartile range, calculated as \(\text{IQR} = Q_3 - Q_1\), describes the spread of the middle 50% of the data and is robust to outliers. This property makes the IQR the basis for a common outlier detection rule: any value falling below \(Q_1 - 1.5 \times \text{IQR}\) or above \(Q_3 + 1.5 \times \text{IQR}\) is flagged as unusual, a method used in box plots. The shape of the distribution also matters: a distribution is skewed when it is asymmetric. In a right-skewed (positive) distribution the tail extends to the right and the mean exceeds the median, while in a left-skewed (negative) distribution the tail extends to the left and the mean is less than the median. Recognizing skewness guides whether the mean or median better represents the typical value.

Probability provides the mathematical framework for reasoning about uncertainty. Every probability problem starts with a sample space, the set of all possible outcomes, and events, which are subsets of that space. The complement of an event A, denoted A' or A^c, contains all outcomes not in A, and the complement rule \(P(A') = 1 - P(A)\) is particularly useful when it is easier to compute the probability of "none" rather than "at least one." For combining probabilities, the addition rule governs unions: \(P(A \cup B) = P(A) + P(B) - P(A \cap B)\), where the intersection term is subtracted to avoid double-counting outcomes that belong to both events.

When two events are mutually exclusive (disjoint), they cannot occur simultaneously, so \(P(A \cap B) = 0\) and the addition rule simplifies to \(P(A \cup B) = P(A) + P(B)\). The multiplication rule governs intersections: \(P(A \cap B) = P(A) \times P(B \mid A)\). If A and B are independent, then \(P(B \mid A) = P(B)\) and the rule reduces to \(P(A \cap B) = P(A) \times P(B)\). Note that mutually exclusive events are generally not independent (except when one has probability zero). Conditional probability, \(P(A \mid B) = P(A \cap B) / P(B)\) with \(P(B) > 0\), updates the probability of an event based on new information.

Bayes' theorem, \(P(A \mid B) = P(B \mid A) \times P(A) / P(B)\), links the posterior probability \(P(A \mid B)\) to the prior \(P(A)\), the likelihood \(P(B \mid A)\), and the marginal probability \(P(B)\). When \(P(B)\) is not directly known, the law of total probability expands the denominator, giving \(P(A \mid B) = P(B \mid A) P(A) / [P(B \mid A) P(A) + P(B \mid A') P(A')]\). A useful alternative formulation expresses results through the likelihood ratio \(P(B \mid A) / P(B \mid A')\), where posterior odds equal prior odds multiplied by the likelihood ratio.

Bayes' theorem corrects the base rate fallacy, the tendency to ignore prior probabilities. For example, with disease prevalence \(P(D) = 0.01\), sensitivity \(P(+ \mid D) = 0.95\), and false positive rate \(P(+ \mid D') = 0.05\), the probability of disease after a positive test is \(P(D \mid +) = (0.95 \times 0.01) / (0.95 \times 0.01 + 0.05 \times 0.99) \approx 0.16\). Even an accurate test applied to a low-prevalence population can leave the probability of disease surprisingly low after a positive result. Finally, when counting outcomes, permutations \(P(n, r) = n! / (n - r)!\) count arrangements where order matters, while combinations \(C(n, r) = n! / [r! (n - r)!]\) count selections where order does not: choosing a president and vice president is a permutation, while choosing a committee is a combination.

A random variable assigns a numerical value to each outcome in a sample space, transforming random experiments into quantities we can analyze. Random variables are either discrete, taking a finite or countable set of values, or continuous, taking any value in an interval. For discrete variables, the probability mass function (PMF) gives \(P(X = x)\) for each possible x; these probabilities must be non-negative and sum to one. For continuous variables, the probability density function (PDF) describes the distribution, and probabilities are obtained as areas under the curve: \(P(a \leq X \leq b) = \int_a^b f(x) \, dx\). Although a PDF can exceed one at a single point, the total area under the curve must equal one. Both types are unified by the cumulative distribution function (CDF), \(F(x) = P(X \leq x)\), which is non-decreasing and ranges from zero to one.

The expected value \(E(X)\) represents the long-run average of a random variable, also called the population mean \(\mu\). For discrete distributions, \(E(X) = \sum x_i P(x_i)\), and for continuous distributions, \(E(X) = \int x f(x) \, dx\). Variance quantifies spread around the mean through the identity \(\text{Var}(X) = E(X^2) - [E(X)]^2\), and important properties include \(\text{Var}(aX + b) = a^2 \text{Var}(X)\) and, for independent variables, \(\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y)\). Variance is always non-negative and equals zero only when X is constant.

Several named distributions appear repeatedly in applications. The binomial distribution models the number of successes in n independent trials, each with the same success probability p, and has PMF \(P(X = k) = \binom{n}{k} p^k (1 - p)^{n - k}\) with mean \(np\) and variance \(np(1 - p)\). A binomial experiment requires a fixed number of trials, two outcomes per trial, constant probability of success, and independence. The Poisson distribution models the count of events in a fixed interval when events occur independently and at a constant average rate \(\lambda\), with PMF \(P(X = k) = \lambda^k e^{-\lambda} / k!\) and both mean and variance equal to \(\lambda\). When n is large and p is small (typically \(n \geq 20\) and \(p \leq 0.05\)), the Poisson with \(\lambda = np\) provides an excellent approximation to the binomial. The geometric distribution gives the number of trials until the first success, with PMF \(P(X = k) = (1 - p)^{k-1} p\), mean \(1/p\), variance \((1 - p) / p^2\), and the memoryless property.

For continuous settings, the uniform distribution assigns equal density to all values in \([a, b]\) with PDF \(f(x) = 1 / (b - a)\), mean \((a + b) / 2\), and variance \((b - a)^2 / 12\). The exponential distribution models waiting times between events in a Poisson process, with PDF \(f(x) = \lambda e^{-\lambda x}\) for \(x \geq 0\), mean \(1/\lambda\), variance \(1/\lambda^2\), and the memoryless property \(P(X > s + t \mid X > s) = P(X > t)\). The normal (Gaussian) distribution is the most important continuous distribution, with PDF \(f(x) = \frac{1}{\sigma \sqrt{2\pi}} e^{-(x - \mu)^2 / (2\sigma^2)}\) and the empirical rule that approximately 68% of data falls within one standard deviation of the mean, 95% within two, and 99.7% within three. The standard normal distribution, with mean zero and standard deviation one, serves as a reference: any normal variable can be standardized using \(Z = (X - \mu) / \sigma\), after which probabilities are read from a z-table or computed by software.

Before drawing conclusions from data, we must understand how sample statistics behave across many possible samples. A sampling distribution is the probability distribution of a statistic, such as the sample mean, computed from all possible samples of a given size drawn from a population. Even though any individual sample varies, the sampling distribution reveals the regular pattern of that variation and is central to statistical inference. It describes how the statistic fluctuates from sample to sample and forms the bridge between descriptive sample results and claims about populations.

The Central Limit Theorem (CLT) describes this pattern for the sample mean. Regardless of the shape of the population from which we sample, the sampling distribution of \(\bar{x}\) approaches a normal distribution with mean \(\mu\) and standard error \(\sigma / \sqrt{n}\) as the sample size grows. Typically, \(n \geq 30\) is sufficient, even for heavily skewed populations, though convergence is faster for symmetric populations. The CLT also applies to sample proportions: \(\hat{p} \sim N\!\left(p, \sqrt{p(1 - p) / n}\right)\) when \(np \geq 10\) and \(n(1 - p) \geq 10\), which makes normal approximation methods available for proportions as well.

The standard error \(SE = \sigma / \sqrt{n}\) quantifies how much the sample mean varies across samples and is derived directly from the CLT. As n increases, the standard error decreases: quadrupling the sample size, for example, halves the standard error. Larger samples therefore produce sampling distributions that are more tightly clustered around the population mean and more closely approximate a normal distribution. The CLT is foundational because it justifies using normal-based methods, including z-tests, t-approximations, and confidence intervals, even when the underlying population is not normally distributed. This is why most parametric statistical procedures rely on a large sample size rather than on the assumption of normality itself.

A confidence interval provides a range of plausible values for an unknown population parameter, constructed from sample data. A 95% confidence interval, for example, does not mean there is a 95% probability that the specific interval contains the parameter. Rather, it means that if we repeated the sampling procedure many times and constructed an interval from each sample, about 95% of those intervals would capture the true parameter. This repeated-sampling interpretation is the correct way to read a confidence level.

The general structure of a confidence interval is the point estimate plus or minus a margin of error. For a population mean when the standard deviation \(\sigma\) is known or the sample is large, the interval is \(\bar{x} \pm z^* \cdot \sigma / \sqrt{n}\), where \(z^*\) is the critical value from the standard normal distribution (1.96 for 95%, 2.576 for 99%). When \(\sigma\) is unknown and must be estimated by the sample standard deviation s, the t-distribution is used instead, giving \(\bar{x} \pm t^* \cdot s / \sqrt{n}\), where \(t^*\) has \(n - 1\) degrees of freedom. For large n, the t critical value approaches the corresponding z value. For a population proportion, the interval is \(\hat{p} \pm z^* \cdot \sqrt{\hat{p}(1 - \hat{p}) / n}\), valid when \(n\hat{p} \geq 10\) and \(n(1 - \hat{p}) \geq 10\).

The margin of error \(E = z^* \cdot \sigma / \sqrt{n}\) represents the maximum expected difference between the sample estimate and the true parameter at the given confidence level. Several factors influence the width of an interval. Increasing the sample size decreases the standard error and narrows the interval, providing more precision. Increasing the confidence level, for example from 95% to 99%, widens the interval because a larger critical value is required, illustrating the trade-off between confidence and precision. There is no free lunch: higher confidence means a wider, less precise interval, while lower confidence yields a narrower but less reliable one.

Hypothesis testing is a formal procedure for deciding whether sample data provides enough evidence to reject a default claim about a population. The null hypothesis \(H_0\) represents the status quo, typically stating that there is no effect or no difference (such as \(\mu = \mu_0\)). The alternative hypothesis \(H_1\) is the researcher's competing claim, which may be one-tailed (for example \(\mu > \mu_0\) or \(\mu < \mu_0\)) or two-tailed (\(\mu \neq \mu_0\)). The significance level \(\alpha\) is set before collecting data and represents the maximum acceptable probability of a Type I error, which occurs when we reject a true null hypothesis. Common choices are 0.05, 0.01, and 0.10, depending on the consequences of a false positive. A Type II error, denoted by \(\beta\), occurs when we fail to reject a false null hypothesis, and the power of a test, \(1 - \beta\), is the probability of correctly rejecting a false null. Power increases with larger sample sizes, larger effect sizes, higher significance levels, and lower variability, with a common target of power at least 0.80.

The p-value is the probability of observing data as extreme as, or more extreme than, the actual sample, assuming \(H_0\) is true; smaller p-values indicate stronger evidence against \(H_0\). The decision rule is to reject \(H_0\) if the p-value is at most \(\alpha\), and otherwise to fail to reject it. Importantly, a small p-value does not prove \(H_1\) is true, nor does it give the probability that \(H_0\) is true. A p-value of 0.05 means there is a 5% probability of obtaining a result as extreme as the observed one if \(H_0\) were true; it does not mean there is a 5% chance that \(H_0\) is true. It is also essential to distinguish statistical significance from practical significance: a large sample can produce a tiny, statistically significant effect that is too small to matter in practice. When many tests are performed at once, the chance of at least one false positive inflates, which is why corrections such as the Bonferroni adjustment (\(\alpha / m\)) or the Benjamini-Hochberg procedure are used.

For testing means, the z-test is appropriate when \(\sigma\) is known and the sample is large (\(n \geq 30\)), with test statistic \(z = (\bar{x} - \mu_0) / (\sigma / \sqrt{n})\). When \(\sigma\) is unknown or the sample is small, the t-test is used, with statistic \(t = (\bar{x} - \mu_0) / (s / \sqrt{n})\) and \(n - 1\) degrees of freedom; the t-distribution has heavier tails than the normal to account for the added uncertainty. To compare two independent groups, the two-sample t-test uses \(t = (\bar{x}_1 - \bar{x}_2) / \sqrt{s_1^2 / n_1 + s_2^2 / n_2}\) and assumes independence, approximate normality, and (for the pooled version) equal variances. The paired t-test analyzes within-pair differences, with \(t = \bar{d} / (s_d / \sqrt{n})\), and is appropriate when the same subjects are measured twice.

For categorical data, the chi-square test of independence examines whether two categorical variables are related by comparing observed and expected frequencies in a contingency table using \(\chi^2 = \sum (O - E)^2 / E\), where the expected count under independence is the row total times the column total divided by the grand total. A large \(\chi^2\) suggests that the variables are not independent. The chi-square goodness-of-fit test uses the same formula to determine whether observed category counts match an expected distribution, for example testing whether a die is fair. Degrees of freedom are \(k - 1\) for goodness of fit with k categories and \((r - 1)(c - 1)\) for independence with r rows and c columns.

The Pearson correlation coefficient \(r\) measures the strength and direction of the linear relationship between two continuous variables, ranging from \(-1\) (perfect negative) through \(0\) (no linear association) to \(+1\) (perfect positive). It is computed as \(r = \sum (x_i - \bar{x})(y_i - \bar{y}) / \sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}\). A high correlation indicates that variables move together, but it does not mean one causes the other, and this distinction is one of the most important in all of statistics.

A confounding variable, a third factor that influences both variables of interest, can create a spurious correlation, an apparent relationship with no direct causal link. For instance, ice cream sales and drowning deaths are positively correlated, but both are driven by hot weather rather than by each other. The ecological fallacy is a related trap in which conclusions about individuals are drawn from group-level data; a finding that countries with higher chocolate consumption have more Nobel laureates does not imply that individual chocolate eaters are more likely to win a prize. Establishing causation requires controlled experiments with random assignment, not merely correlation.

Simple linear regression models the relationship between one independent variable and one dependent variable as a straight line, \(\hat{y} = b_0 + b_1 x\), where the slope \(b_1\) indicates the expected change in y for a one-unit increase in x and \(b_0\) is the y-intercept. The least squares method chooses the line that minimizes \(\sum (y_i - \hat{y}_i)^2\), the sum of squared residuals. The slope is calculated as \(b_1 = \sum (x_i - \bar{x})(y_i - \bar{y}) / \sum (x_i - \bar{x})^2\) and the intercept as \(b_0 = \bar{y} - b_1 \bar{x}\).

R-squared (\(R^2\)) measures the proportion of variance in y explained by the model, ranging from 0 to 1; for simple regression, \(R^2 = r^2\). Multiple regression extends this idea to several predictors as \(\hat{y} = b_0 + b_1 x_1 + b_2 x_2 + \cdots + b_k x_k\), allowing analysis of how multiple variables jointly influence an outcome while controlling for others. Adjusted R-squared penalizes the addition of irrelevant predictors and is preferred when comparing models with different numbers of variables. Residuals \(e_i = y_i - \hat{y}_i\) play a central role in checking model assumptions: randomly scattered residuals suggest a good fit, while patterns indicate problems such as nonlinearity or heteroscedasticity. Key assumptions of linear regression include linearity, independence of residuals, homoscedasticity (constant residual variance), normality of residuals, and no multicollinearity among predictors. Multicollinearity, which arises when independent variables are highly correlated, inflates standard errors and makes coefficient estimates unreliable; it is often detected using the variance inflation factor, with values above 10 considered problematic.

The validity of any statistical conclusion depends critically on how the data was generated, and different sampling methods offer different trade-offs between practicality and representativeness. In simple random sampling, every member of the population has an equal chance of selection, often implemented using random number generators; this is the gold standard for unbiased sampling but may be impractical for large or geographically dispersed populations. Stratified sampling divides the population into homogeneous subgroups, called strata, and randomly samples from each, ensuring that all subgroups are represented; this can increase precision when strata differ on the variable of interest. Cluster sampling divides the population into clusters (often geographic), randomly selects some clusters, and then surveys all or a sample of individuals within them, which is cost-effective for large populations but typically has higher sampling error than simple random or stratified approaches. In systematic sampling, every kth element is selected from a list after a random starting point, which is simple to implement but can produce biased results if the list has a periodic pattern. Convenience sampling, by contrast, recruits whoever is easiest to reach, is prone to selection bias, and generally cannot be generalized to the broader population.

Bias can also arise after sampling. Sampling bias occurs when the sampling method systematically favors certain members of the population, leading to a non-representative sample. Voluntary response bias, in which only motivated or opinionated individuals respond, and undercoverage, in which some segments of the population are missed, are common forms. Response bias occurs when respondents give inaccurate answers because of question wording, social desirability, leading questions, or misunderstanding; careful survey design with neutral wording helps minimize this.

The design of a study determines whether causal conclusions are possible. In a controlled experiment, the researcher manipulates an independent variable while holding other variables constant and observes the effect on a dependent variable, comparing a treatment group with a control group. Random assignment of participants to groups distributes confounding variables evenly, allowing any outcome differences to be attributed to the treatment rather than to other factors. A placebo, an inactive treatment such as a sugar pill given to the control group, accounts for the placebo effect, the phenomenon in which participants improve simply because they believe they are receiving treatment. In a double-blind experiment, neither participants nor researchers know who is in which group, protecting against both participant expectations and researcher bias. By contrast, an observational study measures variables without manipulating them. Although such studies are necessary when experiments are unethical or impractical, the absence of random assignment means they cannot rule out confounding variables and therefore cannot establish causation on their own.