Concept Architecture
Concept
Theoretically, Significance Level is the pre-specified probability of committing a Type I error when performing a statistical hypothesis test. Denoted by �, it represents the maximum acceptable probability of incorrectly rejecting a true null hypothesis. The significance level is determined before analysing the data and forms a fundamental component of the Neyman?Pearson framework for statistical decision-making. It establishes the evidential threshold required for declaring statistical significance.
Mathematically, the significance level defines the rejection region of the sampling distribution under the null hypothesis. A null hypothesis is rejected when the calculated p-value is less than or equal to �, or equivalently when the observed test statistic exceeds the corresponding critical value. The value of � also determines the confidence level, with a confidence level of (1 ? �).
In practice, significance levels are specified during study design and sample size calculation, most commonly using � = 0.05 for two-sided analyses, although more stringent values such as 0.01 may be used where false positive findings have substantial consequences. In health economics, significance levels are applied when evaluating treatment effects, healthcare costs, quality-adjusted life years, regression coefficients, and other clinical or economic outcomes.
Purpose
Used to define the maximum acceptable probability of a Type I error, establish statistical decision thresholds, guide hypothesis testing, determine critical values, and support sample size determination in health economic research.
Mathematical Formulae
Primary Formula
� = P(Reject H? | H? is true)
Supporting Formulae
Decision rule:
Reject H? if p � �
Confidence Level:
Confidence Level = 1 ? �
Critical value for a two-sided z-test:
z* = �??(1 ? �/2)
where �?? is the inverse cumulative distribution function of the standard Normal distribution.
Related Mathematical Methods
- Hypothesis Testing
- Type I Error
- Type II Error
- Statistical Power
- P-value
- Confidence Interval
- Critical Value
- Sample Size Calculation
- Neyman?Pearson Framework
Example
A health economic evaluation compares mean annual healthcare costs between two interventions.
The study specifies:
� = 0.05
The observed p-value from the analysis is:
p = 0.021
Since:
0.021 � 0.05
the null hypothesis is rejected, providing statistically significant evidence of a difference in mean healthcare costs at the 5% significance level.
Excel Implementation
| Function | Example Formula | Health Economics Application |
|---|---|---|
| IF | =IF(B2<=0.05,"Reject H0","Do Not Reject H0") | Apply the decision rule using the specified significance level. |
| NORM.S.INV | =NORM.S.INV(1-0.05/2) | Calculate the critical z-value for a two-sided test. |
| T.INV.2T | =T.INV.2T(0.05,98) | Calculate the critical t-value for hypothesis testing. |
| CHISQ.INV.RT | =CHISQ.INV.RT(0.05,1) | Determine the critical chi-square value. |
| F.INV.RT | =F.INV.RT(0.05,4,95) | Determine the critical F-value for ANOVA or regression testing. |
VBA (Optional)
Automate hypothesis-testing decisions using user-specified significance levels and generate statistical summary reports across multiple analyses.
Sources
- Neyman J, Pearson ES. On the Problem of the Most Efficient Tests of Statistical Hypotheses. Philosophical Transactions of the Royal Society A. 1933.
- Fisher RA. Statistical Methods for Research Workers.
- Casella G, Berger RL. Statistical Inference.
- Altman DG. Practical Statistics for Medical Research.
- Briggs A, Claxton K, Sculpher M. Decision Modelling for Health Economic Evaluation.
- Drummond MF, et al. Methods for the Economic Evaluation of Health Care Programmes.
- NICE Health Technology Evaluation Manual.
Related Concepts (2)
Library
Publications
1
Bayesian Methods in Health Economics — Gianluca Baio, 1st Edition ed., 2012 (Chapman & Hall / CRC Press)
An overview of Bayesian statistical methods for the analysis of health economic data, covering economic evaluation concepts, statistical cost-effectiveness analysis, Bayesian computation and MCMC, and applied health economic evaluation.
BookView source →
Frequently Asked Questions (6)
What is the significance level?
The predetermined probability threshold, usually five percent, used to judge whether a result counts as statistically significant.
Source: Fisher 1925
What does the significance level set before a test is run?
The significance level is a threshold, chosen in advance and usually set at five per cent, that defines how unlikely a result must be under the null hypothesis before it counts as statistically significant. Setting it before the test fixes the acceptable risk of a false positive, the chance of declaring an effect when none exists. A result is called significant when its p-value falls below this level. Fixing the false-positive risk in advance is its role. Kirkwood and Sterne (2003) describe this.
Source: Kirkwood & Sterne 2003
What does the significance level represent?
The significance level represents the probability of a type I error, the chance of rejecting the null hypothesis when it is true, that the researcher is willing to accept, set before the analysis. A significance level of five per cent means accepting a five per cent chance of a false positive. So the significance level represents the accepted risk of wrongly finding an effect, which is why it is chosen in advance and why a smaller significance level means a stricter standard with a lower false positive rate, and it defines the threshold for the p-value and the boundary of the rejection region in the test.
Source: Fisher 1925
How is the significance level chosen?
The significance level is chosen in advance, conventionally at five per cent, though a stricter level such as one per cent may be used when false positives are especially costly, or an adjusted level when multiple tests are conducted. So the significance level is set before the analysis according to the acceptable risk of a false positive and the context, with five per cent a common default but not a rule, which is why it may be made more stringent for confirmatory or high-stakes decisions or adjusted for multiplicity, since the choice reflects a judgement about the tolerable false positive rate for the particular study.
Source: Fisher 1925
How does the significance level relate to the p-value?
The significance level relates to the p-value as the threshold against which it is compared: if the p-value, the probability of data as extreme as observed under the null, is below the significance level, the result is declared statistically significant and the null rejected, otherwise it is not. So the significance level and the p-value work together in the test decision, with the significance level the fixed cut-off and the p-value the computed evidence, and comparing them determines significance, though the p-value is best seen as a continuous measure of evidence rather than reduced to a binary verdict at the threshold.
Source: Fisher 1925
What are the implications of the significance level for errors?
The significance level determines the type I error rate, the chance of a false positive, so a lower significance level reduces false positives but, for a fixed sample size, increases the chance of a type II error, missing a true effect, reflecting a trade-off. So the significance level has implications for both error types, since making it stricter lowers false positives at the cost of power, which is why it is chosen with the consequences of each error in mind and why the sample size is set to achieve adequate power at the chosen significance level, balancing the risks of false positive and false negative conclusions.
Source: Fisher 1925
Trust Record
Verified by Dr Darrin Baines
British health economist
Professional identity: darrinbaines.org
Verification date: 25 Dec 2025
Content version: 1.0.0
Canonical Identity
- Persistent URI
- https://healtheconomics.wiki/concept/significance-level
- Term code
- HE-ES-SA-195
Stable URI · Machine-readable · Resolvable · CC BY 4.0