Statistical Power

What statistical power means

It measures a design's ability to detect an effect of a chosen size; it is not the probability that a statistically significant finding is true.

$$\text{Power}=P(\text{reject }H_0\mid H_1\text{ is true})=1-\beta$$

Here, $\beta$ is the probability of a Type II error. A target power of 80%, for example, means accepting a 20% probability of failing to reject the null hypothesis when the prespecified alternative is true.

The effect a study is powered to detect

Power is always conditional on assumptions, especially the effect size. The design effect should be clinically or policy relevant and plausible, rather than selected merely because it produces an affordable sample size.

For a continuous outcome, the standardised mean difference is:

$$d=\frac{\Delta}{\sigma}$$

where $\Delta$ is the mean difference to be detected and $\sigma$ is the assumed standard deviation. Smaller target effects, greater outcome variability, or stricter error control generally require larger samples.

What determines power

Power depends on the complete analysis and data-generating process, not sample size alone. Assumptions should match the planned estimand, statistical model, follow-up period, and handling of missing or non-adherent participants.

• Effect size: Larger true effects are easier to detect.
• Sample size: More observations usually increase power.
• Outcome variability: Greater unexplained variation reduces power.
• Significance threshold: A smaller $\alpha$ makes rejection harder and reduces power for a fixed sample size.
• Sidedness: A two-sided test generally requires more information than a one-sided test at the same overall $\alpha$.
• Allocation: Equal allocation is usually most efficient when group variances and per-participant costs are similar.
• Design structure: Clustering, repeated measurements, stratification, covariate adjustment, and event timing affect information.
• Multiplicity: Testing several outcomes, comparisons, or interim looks may require adjusted thresholds.
• Data quality: Missing data, crossover, non-adherence, measurement error, and model misspecification can reduce effective power.

A two-group sample-size calculation

For two independent groups with equal allocation, a continuous outcome, common standard deviation $\sigma$, and a two-sided test, an approximate sample size per group is:

$$n=\frac{2\sigma^2\left(z_{1-\alpha/2}+z_{1-\beta}\right)^2}{\Delta^2}$$

Suppose the target difference is $\Delta=5$, the standard deviation is $\sigma=10$, $\alpha=0.05$ two-sided, and target power is 80%. Using $z_{0.975}=1.96$ and $z_{0.80}=0.84$:

$$n=\frac{2(10^2)(1.96+0.84)^2}{5^2} =\frac{200(7.84)}{25} =62.72$$

The result is rounded up to 63 analysable participants per group. Software using an exact small-sample distribution may produce a slightly different result, so the calculation method and assumptions should be reported.

Allowing for attrition

The analysable sample is inflated when some recruited participants are expected to be unavailable for the primary analysis. If $r$ is the anticipated attrition proportion, the recruitment target is:

$$n_{\text{recruit}}=\frac{n_{\text{analyse}}}{1-r}$$

With 10% anticipated attrition, $63/0.90=70$ participants are required per group. This adjustment preserves the planned analysable number only if the attrition assumption is realistic; it does not eliminate bias caused by informative missingness.

Extensions for common study designs

Different outcomes and designs require different information measures. A familiar closed-form formula should not be transferred to another design without checking its assumptions.

For equal cluster size $m$ and intracluster correlation coefficient $\rho$, a simple variance inflation factor is:

$$DE=1+(m-1)\rho$$

The approximate effective sample size is $n_{\text{effective}}=n/DE$. Unequal cluster sizes usually cause additional efficiency loss, so cluster-level design calculations should use appropriate specialist methods.

Simulation-based power

Simulation is useful when the planned analysis is too complex for a reliable closed-form calculation. The simulation must reproduce the intended design, missing-data mechanism, analysis model, decision rule, and all relevant sources of correlation and uncertainty.

For each candidate sample size, generate many datasets under the specified alternative, apply the prespecified analysis, and estimate power as:

$$\widehat{\text{Power}}=\frac{\text{number of simulated datasets rejecting }H_0}{B}$$

where $B$ is the number of simulation replicates. Because this estimate has Monte Carlo error, the number of replicates and a simulation uncertainty interval should be reported.

Precision, confidence intervals, and observed power

A power calculation asks how often a future design would meet a decision rule under specified assumptions, whereas a precision calculation targets the expected width of a confidence interval. Designs should be based on precision when estimation, rather than hypothesis testing, is the principal objective.

After a study is complete, confidence intervals and effect estimates are more informative than “observed” or post hoc power calculated from the observed effect. Such observed power is largely a transformation of the study's $p$-value and does not repair an imprecise result.

Interim review and sample-size reassessment

Blinded nuisance-parameter reassessment can update assumptions such as outcome variance or aggregate event rate without estimating the treatment effect. Unblinded reassessment, conditional power, predictive power, or early stopping can alter operating characteristics and should follow a prespecified, independently governed design.

Any adaptive rule must preserve the intended Type I error rate and account for operational bias. Conditional power conditions on current data and an assumed future effect, whereas predictive power averages over uncertainty about that future effect; the two quantities should not be treated as interchangeable.

Relevance to health economics

A trial powered for a clinical endpoint is not automatically powered for costs, quality-adjusted life-years, net benefit, or subgroup cost-effectiveness. Economic outcomes may be skewed, correlated, censored, missing, and driven by rare high-cost events, so joint or simulation-based design work may be needed.

Statistical power also differs from decision uncertainty. A study can lack conventional power for one hypothesis yet still change a reimbursement decision, while value-of-information analysis asks whether reducing uncertainty is worth the cost of further research.

A practical workflow

Power analysis is most defensible when it is developed alongside the protocol and statistical analysis plan. The following sequence makes assumptions, calculations, and design consequences auditable.

1. Define the population, intervention, comparator, outcome, follow-up, estimand, and primary analysis.
2. Specify the null and alternative hypotheses and whether the test is one-sided or two-sided.
3. Choose a justified significance level and target power, including any multiplicity adjustment.
4. Set the smallest relevant effect and support nuisance parameters with suitable evidence.
5. Represent allocation, clustering, repeated observations, censoring, accrual, and other design features.
6. Calculate the required analysable sample or event count using validated software or simulation.
7. Inflate for anticipated attrition, non-adherence, missingness, or design inefficiency without double counting.
8. Stress-test uncertain assumptions with sensitivity analyses and plausible scenarios.
9. Document formulas, software, settings, inputs, rounding, and the sample-size consequences of each assumption.
10. Align the final calculation with the protocol, analysis plan, recruitment target, and reporting materials.

Interpretation and reporting

Power is a long-run operating characteristic under a specified alternative, not a guarantee about the result of one study. A transparent report states the primary outcome and analysis, target effect, nuisance parameters, $\alpha$, target power, sidedness, allocation, design adjustments, attrition allowance, calculation method, software, and final rounding.

Common errors include powering on an implausibly large effect, confusing absence of significance with equivalence, ignoring clustering or multiplicity, adding attrition incorrectly, and presenting a sample-size number without its assumptions. Sensitivity analyses are especially important when variance, event rates, intracluster correlation, or attrition are uncertain.