Hypothesis Testing

Hypothesis Testing: Concept Architecture

Orientation and learning roadmap

Hypothesis testing provides a structured way to evaluate evidence about a population quantity while controlling specified long-run error rates. This page develops the null and alternative hypotheses, test statistic, sampling distribution, p-value, rejection rule, error rates, power, and confidence-interval connection. It then distinguishes superiority, non-inferiority, equivalence, multiplicity, clinical importance, and decision relevance in health research and economic evaluation.

A test begins with a precise population claim

A statistical hypothesis is a statement about a population parameter, data-generating process, or probability distribution. The null hypothesis (H_0) defines the claim tested directly, while the alternative (H_1) defines the competing claim supported when the data are sufficiently incompatible with (H_0). Both should be specified before outcome-dependent analysis choices are made.

The null hypothesis is not always “no effect”

In a conventional superiority test, the null may state that a treatment difference is zero or unfavorable. Non-inferiority and equivalence designs use null hypotheses defined by clinically justified margins rather than exact equality. Interpreting a result requires the actual mathematical hypotheses, not just the labels “positive” or “negative.”

One-sided and two-sided tests answer different questions

A two-sided alternative allows departures in either direction, while a one-sided alternative considers a prespecified direction. Choosing one side after viewing the estimate invalidates the intended error control. A one-sided test is appropriate only when effects in the opposite direction would not support the claim and safety remains assessed independently.

The test statistic orders possible data outcomes

A test statistic summarizes how far the observed estimate lies from the null value relative to its expected variability. Its scale and direction determine which possible outcomes count as more incompatible with the null model. For an approximately normal estimator (\widehat{\theta}) with standard error (SE(\widehat{\theta})), a common statistic is:

[ Z=\frac{\widehat{\theta}-\theta_0}{SE(\widehat{\theta})}, ]

where (\theta_0) is the value specified under the null hypothesis.

The reference distribution depends on assumptions

The null distribution describes how the statistic would vary across repeated samples if the null model and all relevant assumptions were correct. Normal, (t), chi-square, (F), exact, permutation, and bootstrap-based tests use different reference mechanisms. Independence, distributional form, variance structure, clustering, censoring, model specification, and sampling design can all affect validity.

The p-value is conditional on the null model

The p-value is the probability, under the null hypothesis and the test’s assumptions, of obtaining a result at least as incompatible with the null as the observed result according to the chosen statistic. It is not the probability that the null hypothesis is true, the probability that the result occurred “by chance,” or the size of the effect. For a two-sided standard-normal test:

[ p=2\left[1-\Phi\left(|Z_{\mathrm{obs}}|\right)\right]. ]

The significance level controls a long-run error rate

The significance level (\alpha) is a prespecified upper bound on the probability of rejecting the null when it is true under the testing procedure. Comparing (p) with (\alpha) implements a decision rule, but it does not turn evidence into certainty. The conventional rule is:

[ \text{Reject }H_0\text{ when }p\leq\alpha. ]

Type I and Type II errors describe different failures

A Type I error occurs when the test rejects a true null hypothesis, while a Type II error occurs when it fails to reject a false null hypothesis at a specified alternative. The Type II error probability is denoted (\beta), and power is (1-\beta). These probabilities describe repeated use of the procedure under defined parameter values; they are not posterior probabilities that a particular conclusion is wrong.

Power depends on a specific alternative

Power is the probability that a test rejects the null when a particular alternative effect is true. It depends on effect size, variability, sample size, allocation, significance level, test direction, design effects, missingness, nonadherence, and analysis method. Reporting that a study has “80% power” is incomplete without stating the effect and assumptions used.

Sample size planning should target meaningful effects

Prospective sample-size calculations usually choose a clinically or policy-relevant effect, desired power, significance level, variability, and design structure. Inflating the sample for attrition does not address bias from informative missingness or poor measurement. Very large studies can detect negligible effects, while small studies can miss important effects and produce unstable estimates.

Worked test of a mean difference

Suppose a study estimates a mean treatment difference of 4.0 units with a standard error of 1.5 units and tests (H_0:\theta=0) against a two-sided alternative. The statistic is (Z=4/1.5=2.6667), giving a two-sided normal-approximation p-value of approximately 0.0077. The result is incompatible with the null at (\alpha=0.05), but its practical importance still depends on the outcome scale and decision context.

[ Z=\frac{4.0-0}{1.5}=2.6667, ]

[ p=2\left[1-\Phi(2.6667)\right]\approx0.0077. ]

Confidence intervals show magnitude and precision

For many standard procedures, a two-sided hypothesis test at level (\alpha) rejects a point null exactly when the corresponding (100(1-\alpha)%) confidence interval excludes that null value. The interval displays a range of parameter values compatible with the data and model at the stated level. It is not a probability statement that the fixed parameter lies inside the realized interval under ordinary frequentist interpretation.

For the worked example, an approximate 95% confidence interval is:

[ 4.0\pm1.96(1.5)=[1.06,6.94]. ]

Failure to reject is not proof of no effect

A nonsignificant result means the test did not produce sufficient evidence to reject the specified null at the chosen threshold. It may reflect a small effect, imprecision, high variability, weak design, model misspecification, or an effect in an unexpected direction. Demonstrating similarity requires an equivalence or non-inferiority design with justified margins, not a nonsignificant superiority test.

Superiority tests ask whether an option is better

A superiority test evaluates whether the effect differs from, or exceeds, a comparator under prespecified direction and error control. Statistical superiority does not establish that the effect is clinically important, cost-effective, affordable, or transferable. Estimates and confidence intervals should accompany the testing decision.

Non-inferiority tests protect against an unacceptable loss

A non-inferiority test evaluates whether a new intervention is not worse than a comparator by more than a prespecified margin (\Delta). The margin must preserve clinically meaningful benefit and be justified independently of the observed data. Bias toward similarity, nonadherence, treatment switching, and analysis-population choices can make non-inferiority claims especially vulnerable.

Equivalence requires evidence within two margins

An equivalence test asks whether the entire plausible effect lies within a prespecified interval ((-\Delta,\Delta)). The two one-sided tests procedure evaluates both boundaries, and equivalence is concluded only when both null components are rejected at the stated level. “No statistically significant difference” is not evidence of equivalence.

Multiple testing changes the chance of false claims

Testing many endpoints, time points, subgroups, models, or interim looks creates multiple opportunities to reject a true null. Family-wise error, false discovery rate, gatekeeping, hierarchical testing, alpha spending, and simultaneous inference address different goals. The multiplicity strategy should be prespecified and aligned with which claims are confirmatory.

Data-dependent analysis can invalidate nominal p-values

Selecting outcomes, covariates, subgroups, transformations, stopping times, or models after viewing results changes the effective testing procedure. A nominal p-value from the selected analysis does not account for the unreported alternatives considered. Protocols, analysis plans, registration, versioned code, and complete reporting make the actual evidence pathway visible.

Model assumptions belong to the claim

A p-value is valid only relative to the statistical model and sampling process used to derive it. Heteroskedasticity, clustering, repeated measures, missing data, informative censoring, nonproportional hazards, overdispersion, and nonlinearity may require adapted models or robust inference. Assumption checks should be relevant to the estimand and not reduced to a ritual list of preliminary tests.

Randomization tests and observational tests differ in interpretation

In randomized experiments, valid randomization-based tests can ground inference in the treatment-allocation mechanism. In observational data, hypothesis tests rely more heavily on modelling and identification assumptions, including confounding control and selection. A small p-value cannot repair an unidentified causal effect.

Subgroup claims require interaction tests

Evidence that an effect is significant in one subgroup but not another does not establish that subgroup effects differ. The relevant test evaluates the interaction or contrast between subgroup-specific effects. Low power, multiplicity, and post hoc subgroup definitions make these analyses especially prone to exaggerated conclusions.

Statistical significance is not clinical importance

The p-value reflects compatibility with a null model, not the size, benefit, harm, or value of an effect. Clinical thresholds, minimally important differences, absolute risks, adverse outcomes, patient preferences, costs, and equity consequences require separate evaluation. A precise trivial effect can be statistically significant, and an important uncertain effect can be nonsignificant.

Economic decisions should not be reduced to null-hypothesis tests

Cost-effectiveness decisions concern expected incremental costs and outcomes under uncertainty, not whether each component is individually statistically significant. Requiring statistically significant differences in costs or effects before modelling can discard relevant evidence and bias decisions. Probabilistic analysis, expected net benefit, and value-of-information methods are usually more directly aligned with decision uncertainty.

Bayesian analyses answer different probability questions

Bayesian methods combine a likelihood with a prior distribution to produce posterior probabilities and credible intervals. A posterior probability that an effect exceeds a threshold is not a frequentist p-value, even when the numerical values appear similar. The prior, model, estimand, and decision loss should be reported so the inference can be interpreted correctly.

Sequential and adaptive analyses need planned error control

Repeatedly examining accumulating data and stopping when significance appears inflates false-positive risk under an ordinary fixed-sample test. Group-sequential and adaptive designs use prespecified boundaries, alpha-spending functions, or other valid procedures to preserve the intended operating characteristics. Unplanned optional stopping cannot be repaired by reporting the final nominal p-value alone.

Excel can reproduce a simple test transparently

A worksheet can display the estimate, null value, standard error, statistic, p-value, confidence limits, and decision threshold in separate labeled cells. If the estimate is in B2, null value in B3, and standard error in B4, the statistic is =(B2-B3)/B4 and a two-sided standard-normal p-value is =2*(1-NORM.S.DIST(ABS(B5),TRUE)). Complex survey, clustered, survival, longitudinal, adaptive, or multiplicity-adjusted analyses require methods suited to their design.

A disciplined hypothesis-testing workflow

Valid testing begins with the scientific question and ends with a calibrated interpretation of estimates, uncertainty, and decision consequences. Each analytical choice should be connected to the estimand and study design before results are known. A practical sequence is:

1. Define the estimand. Specify the population, outcome, treatment condition, summary measure, and handling of intercurrent events.
2. State both hypotheses. Write the null and alternative mathematically, including direction and any margin.
3. Set error criteria. Prespecify (\alpha), desired power, and the family of claims requiring multiplicity control.
4. Choose the design and test. Align randomization, clustering, repeated observations, censoring, and distributional assumptions.
5. Plan sample size. Use a meaningful alternative effect and realistic variability, attrition, and design effects.
6. Protect analysis integrity. Register outcomes, models, subgroups, interim rules, and deviations.
7. Calculate and diagnose. Verify data, assumptions, statistic, reference distribution, and numerical implementation.
8. Report estimates and intervals. Present magnitude and precision alongside p-values and adjusted analyses.
9. Interpret in context. Separate statistical evidence from clinical importance, bias, causality, and economic value.
10. Preserve uncertainty. Avoid converting a threshold crossing into certainty or a nonsignificant result into proof of no effect.

Common interpretation errors

Most hypothesis-testing errors arise when a conditional probability is given a stronger meaning than it has or when the tested procedure differs from the reported one. Transparent planning and complete reporting make these errors easier to detect. Common failures include:

• Interpreting the p-value as the probability that the null hypothesis is true.
• Describing (p>0.05) as proof that there is no effect.
• Treating statistical significance as evidence of clinical or economic importance.
• Choosing a one-sided test after seeing the direction of the result.
• Comparing subgroup p-values instead of testing the interaction.
• Ignoring multiple endpoints, models, time points, or interim analyses.
• Reporting only a threshold decision without the effect estimate and confidence interval.
• Using a test whose assumptions do not match the design or data.
• Treating an observational association as causal because its p-value is small.
• Confusing frequentist confidence with Bayesian posterior probability.

What a complete testing record should disclose

A reproducible record identifies the estimand, hypotheses, test direction, significance level, power target, sample-size assumptions, analysis population, statistical model, test statistic, reference distribution, multiplicity strategy, missing-data handling, interim rules, estimate, confidence interval, exact p-value, diagnostics, software, and deviations from plan. It should distinguish confirmatory analyses from exploratory findings and explain practical importance without relying on a significance label. This record allows a reviewer to evaluate both the calculation and the claim it is being used to support.