Hypothesis Testing

A formal framework for evaluating whether observed data provide enough evidence to reject a null hypothesis in favour of an alternative.

Source: Neyman & Pearson 1933

Hypothesis testing sets out to answer whether the data provide enough evidence to reject a null hypothesis of no effect in favour of an alternative that there is one. It works by asking how likely the observed result, or one more extreme, would be if the null were true, and rejecting the null when that likelihood falls below a chosen threshold. This gives a formal, if blunt, rule for deciding whether an apparent effect is more than chance. Judging evidence against a no-effect assumption is its purpose. Kirkwood and Sterne (2003) describe this framework.

Source: Kirkwood & Sterne 2003

Hypothesis testing works by specifying a null and an alternative hypothesis, choosing a significance level, computing a test statistic from the data, and determining how extreme it is under the null, either by comparing it with a critical value or by calculating a p-value. If the result is sufficiently extreme, the null is rejected. So hypothesis testing works by quantifying how compatible the data are with the null hypothesis and rejecting the null when they are sufficiently incompatible, using the significance level to control the risk of a false rejection, which yields a decision about whether the evidence supports the alternative.

Source: Neyman & Pearson 1933

In hypothesis testing, a type I error is rejecting the null hypothesis when it is actually true, a false positive, whose probability is the significance level, alpha; a type II error is failing to reject the null when it is false, a false negative, whose probability is beta, with power being one minus beta. So the two errors represent the ways a test can be wrong, with the significance level controlling the false positive rate and the power controlling the ability to detect a true effect, and both are considered in designing a test, since there is a trade-off between them for a fixed sample size.

Source: Neyman & Pearson 1933

The p-value in hypothesis testing is the probability of obtaining a result as extreme as, or more extreme than, the observed one if the null hypothesis were true; a small p-value indicates the data are unlikely under the null, providing evidence against it. If the p-value is below the significance level, the null is rejected. So the p-value quantifies the compatibility of the data with the null hypothesis, serving as the basis for the test decision, though it is often misinterpreted, as it is not the probability that the null is true, and it is best understood as a measure of evidence against the null under the assumptions of the test.

Source: Neyman & Pearson 1933

The criticisms of hypothesis testing include that p-values are frequently misinterpreted, for example as the probability the null is true; that the dichotomy of significant versus non-significant encourages crude thinking and neglect of effect size; that a chosen significance level is arbitrary; and that significance depends on sample size. So hypothesis testing is used with awareness of these criticisms, since focusing on a yes-or-no verdict can obscure the magnitude and precision of effects, which is why confidence intervals and effect sizes are recommended alongside or instead of bare significance tests, to give a fuller and less misleading account of what the data show.

Source: Neyman & Pearson 1933