The validity of any statistical conclusion depends critically on how the data was generated, and different sampling methods offer different trade-offs between practicality and representativeness. In simple random sampling, every member of the population has an equal chance of selection, often implemented using random number generators; this is the gold standard for unbiased sampling but may be impractical for large or geographically dispersed populations. Stratified sampling divides the population into homogeneous subgroups, called strata, and randomly samples from each, ensuring that all subgroups are represented; this can increase precision when strata differ on the variable of interest. Cluster sampling divides the population into clusters (often geographic), randomly selects some clusters, and then surveys all or a sample of individuals within them, which is cost-effective for large populations but typically has higher sampling error than simple random or stratified approaches. In systematic sampling, every kth element is selected from a list after a random starting point, which is simple to implement but can produce biased results if the list has a periodic pattern. Convenience sampling, by contrast, recruits whoever is easiest to reach, is prone to selection bias, and generally cannot be generalized to the broader population.
Bias can also arise after sampling. Sampling bias occurs when the sampling method systematically favors certain members of the population, leading to a non-representative sample. Voluntary response bias, in which only motivated or opinionated individuals respond, and undercoverage, in which some segments of the population are missed, are common forms. Response bias occurs when respondents give inaccurate answers because of question wording, social desirability, leading questions, or misunderstanding; careful survey design with neutral wording helps minimize this.
The design of a study determines whether causal conclusions are possible. In a controlled experiment, the researcher manipulates an independent variable while holding other variables constant and observes the effect on a dependent variable, comparing a treatment group with a control group. Random assignment of participants to groups distributes confounding variables evenly, allowing any outcome differences to be attributed to the treatment rather than to other factors. A placebo, an inactive treatment such as a sugar pill given to the control group, accounts for the placebo effect, the phenomenon in which participants improve simply because they believe they are receiving treatment. In a double-blind experiment, neither participants nor researchers know who is in which group, protecting against both participant expectations and researcher bias. By contrast, an observational study measures variables without manipulating them. Although such studies are necessary when experiments are unethical or impractical, the absence of random assignment means they cannot rule out confounding variables and therefore cannot establish causation on their own.