Hypothesis testing is a formal procedure for deciding whether sample data provides enough evidence to reject a default claim about a population. The null hypothesis \(H_0\) represents the status quo, typically stating that there is no effect or no difference (such as \(\mu = \mu_0\)). The alternative hypothesis \(H_1\) is the researcher's competing claim, which may be one-tailed (for example \(\mu > \mu_0\) or \(\mu < \mu_0\)) or two-tailed (\(\mu \neq \mu_0\)). The significance level \(\alpha\) is set before collecting data and represents the maximum acceptable probability of a Type I error, which occurs when we reject a true null hypothesis. Common choices are 0.05, 0.01, and 0.10, depending on the consequences of a false positive. A Type II error, denoted by \(\beta\), occurs when we fail to reject a false null hypothesis, and the power of a test, \(1 - \beta\), is the probability of correctly rejecting a false null. Power increases with larger sample sizes, larger effect sizes, higher significance levels, and lower variability, with a common target of power at least 0.80.
The p-value is the probability of observing data as extreme as, or more extreme than, the actual sample, assuming \(H_0\) is true; smaller p-values indicate stronger evidence against \(H_0\). The decision rule is to reject \(H_0\) if the p-value is at most \(\alpha\), and otherwise to fail to reject it. Importantly, a small p-value does not prove \(H_1\) is true, nor does it give the probability that \(H_0\) is true. A p-value of 0.05 means there is a 5% probability of obtaining a result as extreme as the observed one if \(H_0\) were true; it does not mean there is a 5% chance that \(H_0\) is true. It is also essential to distinguish statistical significance from practical significance: a large sample can produce a tiny, statistically significant effect that is too small to matter in practice. When many tests are performed at once, the chance of at least one false positive inflates, which is why corrections such as the Bonferroni adjustment (\(\alpha / m\)) or the Benjamini-Hochberg procedure are used.
For testing means, the z-test is appropriate when \(\sigma\) is known and the sample is large (\(n \geq 30\)), with test statistic \(z = (\bar{x} - \mu_0) / (\sigma / \sqrt{n})\). When \(\sigma\) is unknown or the sample is small, the t-test is used, with statistic \(t = (\bar{x} - \mu_0) / (s / \sqrt{n})\) and \(n - 1\) degrees of freedom; the t-distribution has heavier tails than the normal to account for the added uncertainty. To compare two independent groups, the two-sample t-test uses \(t = (\bar{x}_1 - \bar{x}_2) / \sqrt{s_1^2 / n_1 + s_2^2 / n_2}\) and assumes independence, approximate normality, and (for the pooled version) equal variances. The paired t-test analyzes within-pair differences, with \(t = \bar{d} / (s_d / \sqrt{n})\), and is appropriate when the same subjects are measured twice.
For categorical data, the chi-square test of independence examines whether two categorical variables are related by comparing observed and expected frequencies in a contingency table using \(\chi^2 = \sum (O - E)^2 / E\), where the expected count under independence is the row total times the column total divided by the grand total. A large \(\chi^2\) suggests that the variables are not independent. The chi-square goodness-of-fit test uses the same formula to determine whether observed category counts match an expected distribution, for example testing whether a die is fair. Degrees of freedom are \(k - 1\) for goodness of fit with k categories and \((r - 1)(c - 1)\) for independence with r rows and c columns.