The Pearson correlation coefficient \(r\) measures the strength and direction of the linear relationship between two continuous variables, ranging from \(-1\) (perfect negative) through \(0\) (no linear association) to \(+1\) (perfect positive). It is computed as \(r = \sum (x_i - \bar{x})(y_i - \bar{y}) / \sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}\). A high correlation indicates that variables move together, but it does not mean one causes the other, and this distinction is one of the most important in all of statistics.
A confounding variable, a third factor that influences both variables of interest, can create a spurious correlation, an apparent relationship with no direct causal link. For instance, ice cream sales and drowning deaths are positively correlated, but both are driven by hot weather rather than by each other. The ecological fallacy is a related trap in which conclusions about individuals are drawn from group-level data; a finding that countries with higher chocolate consumption have more Nobel laureates does not imply that individual chocolate eaters are more likely to win a prize. Establishing causation requires controlled experiments with random assignment, not merely correlation.
Simple linear regression models the relationship between one independent variable and one dependent variable as a straight line, \(\hat{y} = b_0 + b_1 x\), where the slope \(b_1\) indicates the expected change in y for a one-unit increase in x and \(b_0\) is the y-intercept. The least squares method chooses the line that minimizes \(\sum (y_i - \hat{y}_i)^2\), the sum of squared residuals. The slope is calculated as \(b_1 = \sum (x_i - \bar{x})(y_i - \bar{y}) / \sum (x_i - \bar{x})^2\) and the intercept as \(b_0 = \bar{y} - b_1 \bar{x}\).
R-squared (\(R^2\)) measures the proportion of variance in y explained by the model, ranging from 0 to 1; for simple regression, \(R^2 = r^2\). Multiple regression extends this idea to several predictors as \(\hat{y} = b_0 + b_1 x_1 + b_2 x_2 + \cdots + b_k x_k\), allowing analysis of how multiple variables jointly influence an outcome while controlling for others. Adjusted R-squared penalizes the addition of irrelevant predictors and is preferred when comparing models with different numbers of variables. Residuals \(e_i = y_i - \hat{y}_i\) play a central role in checking model assumptions: randomly scattered residuals suggest a good fit, while patterns indicate problems such as nonlinearity or heteroscedasticity. Key assumptions of linear regression include linearity, independence of residuals, homoscedasticity (constant residual variance), normality of residuals, and no multicollinearity among predictors. Multicollinearity, which arises when independent variables are highly correlated, inflates standard errors and makes coefficient estimates unreliable; it is often detected using the variance inflation factor, with values above 10 considered problematic.