Correlation quantifies the strength and direction of a linear relationship between two numerical variables. Pearson's correlation coefficient \(r\) ranges from \(-1\) to \(+1\); values near \(0\) indicate no linear association, although a non-linear relationship may still exist. Spearman's correlation captures monotonic (rather than strictly linear) association by working with ranks. Importantly, correlation does not imply causation — apparent associations can arise from coincidence, confounding variables that influence both predictors and outcomes, or reverse causation. Simple linear regression models a straight-line relationship \(\hat{y} = b_0 + b_1 x\) between a single independent variable and a dependent variable. The least squares method chooses the coefficients that minimize the sum of squared residuals, where a residual is the difference \(y - \hat{y}\) between an observed and predicted value. R-squared measures the proportion of variation in the dependent variable explained by the model; for example, \(R^2 = 0.8\) means that 80% of the variability in the response is captured.
Multiple regression extends this idea to two or more independent variables, where each regression coefficient is interpreted as the expected change in \(y\) for a one-unit increase in that predictor, holding other variables constant. Categorical predictors enter the model as dummy variables, with one level designated as the reference category against which the others are compared. Interaction terms, written as products \(x_1 \cdot x_2\), capture how one predictor's effect depends on the level of another. Logistic regression handles a binary outcome by passing a linear combination of predictors through the logistic function to produce a probability; the logit link \(\log(p/(1-p))\) linearizes this relationship, and exponentiating a coefficient yields an odds ratio, the multiplicative change in odds for a one-unit increase in the predictor. Several diagnostic issues can undermine a regression: multicollinearity arises when independent variables are highly correlated and destabilizes coefficient estimates; heteroscedasticity, often visible as a funnel shape in a residual plot of residuals versus fitted values, signals non-constant error variance; outliers, leverage points (extreme x-values), and influential points (those whose removal substantially changes the fit) are flagged using measures such as Cook's distance.
Choosing among models involves the bias-variance tradeoff: simple models underfit and miss structure, while complex models overfit and capture noise, leading to poor generalization. Cross-validation estimates true predictive performance by splitting data into training and validation folds; in k-fold cross-validation this is repeated for k rotations. Regularization discourages overfitting by penalizing complexity: L1 (Lasso) adds a penalty proportional to \(|\beta|\) and can shrink coefficients exactly to zero, performing feature selection, while L2 (Ridge) adds a penalty proportional to \(\beta^2\) and shrinks coefficients toward zero without eliminating them. Information criteria provide further tools for model selection: AIC penalizes complexity gently via \(2k - 2\ln(L)\), while BIC's \(k \ln(n) - 2\ln(L)\) penalizes it more heavily. Other regression diagnostics include Q-Q plots, which compare sample quantiles to those of a reference distribution to check normality, and the Shapiro-Wilk test, which formally tests the null hypothesis of normality.