Signature
d = p_T - p_C; SE_d = sqrt(p_T * (1 - p_T) / n_T + p_C * (1 - p_C) / n_C); L_d = d - z * SE_d; U_d = d + z * SE_d; NI = (L_d > -M); SUP = (L_d > 0)
| Inputs | Definition | Unit |
|---|---|---|
p_T | Proportion of patients in the test arm with the favourable outcome, for example cure, over the trial period | proportion |
p_C | Proportion of patients in the active control arm with the same favourable outcome over the same period | proportion |
n_T | Number of patients analysed in the test arm, above zero | patients |
n_C | Number of patients analysed in the active control arm, above zero | patients |
z | Standard normal critical value for the two-sided interval, 1.96 for 95%, which matches a one-sided test at 0.025 | none |
M | Largest loss on the favourable outcome that is acceptable for the test treatment, fixed in the protocol, entered as a positive number on the same scale as d and not greater than M1, the effect of the active control assumed present in the trial | probability difference |
d | Difference in the proportion of patients with the favourable outcome, test arm minus active control arm, positive when the test treatment does better | probability difference, at least -1 and at most 1 |
|---|---|---|
SE_d | Estimated standard error of d, the square root of p_T (1 minus p_T) over n_T plus p_C (1 minus p_C) over n_C | probability difference |
L_d | Lower limit of the two-sided interval for d, the value read against minus the margin and against zero | probability difference |
U_d | Upper limit of the two-sided interval for d; a value below zero shows the test treatment statistically worse even when NI is 1 | probability difference |
NI | Equals 1 (true) when L_d lies above minus M, so non-inferiority is shown, and 0 (false) otherwise | binary indicator |
SUP | Equals 1 (true) when L_d lies above zero, so superiority of the test treatment is shown, and 0 (false) otherwise | binary indicator |
Function
Non-inferiority and superiority decision function for an active control trial
Maps the proportions of patients with a favourable outcome in the test and active control arms of a two-arm randomised trial, the arm sizes, a normal critical value and a margin fixed in the protocol to the two-sided interval for the risk difference and to the readings for non-inferiority and superiority. The steps before the decision are live records. The risk difference d is the test proportion minus the control proportion, the difference of HE-FM-ARR-001 with its sign reversed, because HE-FM-ARR-001 subtracts the intervention risk of an adverse event from the comparator risk and HE-AS-ARR-002 reverses the subtraction for a beneficial outcome. Each arm's standard error is HE-FM-BINOM-004, the variance of a difference of two independent arm estimates is the sum of their variances as in HE-VR-AN-002 and HE-FM-NMA-001, and the interval limits are the Wald limits of HE-FM-AN-001. HE-FM-ACTR-001 adds the comparison of the lower limit with minus the margin and with zero. The article's economic check is a citation of live records, the incremental net health benefit being HE-FM-CET-002, its break-even shortfall in cures being HE-FM-CET-002 solved for the shortfall under the article's illustrative linear cost and QALY model (compare the break-even price of HE-FM-BEA-001), and the probability of a positive net health benefit following the z statistic of HE-FM-AN-004. Bioequivalence on the ratio scale is HE-FM-ANDA-001, and the least-cost choice once outcomes are shown to be equivalent is HE-FM-CMA-001. Notation follows the Active Control Trial article.
Try this function
Implementations
Excel
Risk difference interval and margin readings in named cells
With the inputs in cells named CureProportionTest, CureProportionControl, ArmSizeTest, ArmSizeControl, CriticalValueZ and NonInferiorityMargin, the six formulas return, in order, cells named RiskDifference, StandardErrorRiskDifference, LowerLimitRiskDifference, UpperLimitRiskDifference, NonInferiorityShown and SuperiorityShown. CriticalValueZ can be entered as 1.96 or as =NORM.S.INV(0.975). A blank ArmSizeTest or ArmSizeControl is treated as 0 and the standard error returns #DIV/0!. A blank NonInferiorityMargin is also treated as 0 but raises no error, so NonInferiorityShown then silently returns the superiority reading.
=CureProportionTest-CureProportionControl; =SQRT(CureProportionTest*(1-CureProportionTest)/ArmSizeTest+CureProportionControl*(1-CureProportionControl)/ArmSizeControl); =RiskDifference-CriticalValueZ*StandardErrorRiskDifference; =RiskDifference+CriticalValueZ*StandardErrorRiskDifference; =IF(LowerLimitRiskDifference>-NonInferiorityMargin,1,0); =IF(LowerLimitRiskDifference>0,1,0)
Assumptions
Margin fixed in the protocol and not greater than the control effect
M is chosen and justified before the data are seen, as CPMP/EWP/482/99 section III and the EMEA 2005 margin guideline section 2 require, because a margin chosen after the results can always be made to give non-inferiority. ICH E10 section 1.5.1.1 states that the margin cannot be greater than the smallest effect the active control would reliably be expected to have against placebo in the planned trial, and the FDA guidance sets M2, the largest clinically acceptable loss, at or below M1. The EMEA 2005 guideline section 3 asks for a conservative margin when constancy of the control effect cannot be assured. In the article's illustrative example M1 is 0.22 and M2 is 0.10 on the risk difference scale.
Favourable outcome and test minus control orientation
d is the test proportion minus the control proportion of a favourable outcome, so values above zero favour the test treatment, the orientation of ICH E9 section 3.3.2 (investigational product minus control) and of CPMP/EWP/482/99 section II.1. For an adverse outcome such as death, d is entered as the control risk minus the test risk, which is the ARR of HE-FM-ARR-001, and the same rule then applies (derived). M is entered as a positive number in both cases.
Independent randomised arms large enough for the Wald interval
The two arms are independent randomised groups, so the variance of d is the sum of the arm variances, and each arm is large enough, with its proportion away from 0 and 1, for the normal approximation of HE-FM-AN-001 to hold (compare HE-AS-BINOM-009 for one proportion). Newcombe (1998) found that computationally simpler intervals for a difference of independent proportions are prone to aberrations and poor coverage, and that the methods of Mee and of Miettinen and Nurminen and a method combining Wilson score intervals perform well, so a margin decision close to the boundary in a small trial is better checked with one of those intervals (derived).
Two-sided 95% interval and a strict inequality at the margin
z is 1.96, the two-sided 95% interval that CPMP/EWP/482/99 section II.4 assumes for all clinical trials whatever their objective, with a one-sided interval at 97.5% as its counterpart. The FDA guidance section III.C states that the two-sided 95% interval corresponds to a one-sided test size of 0.025, and ICH E9 section 5.5 prefers one-sided type I errors at half the two-sided level. The FDA guidance section III.A establishes non-inferiority when the upper bound for control minus test is less than M, and shows an effect over placebo when the lower bound is above 0, so a lower limit L_d exactly equal to minus M gives NI equal to 0 and L_d exactly equal to zero gives SUP equal to 0 (derived).
Worked examples
Article's antibiotic trial read against a 10-point margin
The article's hypothetical trial randomises 500 patients to each arm, with cure in 84% on test drug T and 86% on active control C, and a clinical margin M2 of 10 points. The difference is minus 0.02 with standard error 0.0226, and the 95% interval has a lower limit of minus 0.0642 and an upper limit of plus 0.0242. The lower limit lies above minus 0.10, so non-inferiority is shown, and it also lies above minus 0.22, the article's M1. The interval contains zero, so superiority is not shown. All figures are the article's illustrative inputs.
p_T = 0.84; p_C = 0.86; n_T = 500; n_C = 500; z = 1.96; M = 0.10; d = -0.02; SE_d = 0.0226; L_d = -0.0642; U_d = 0.0242; NI = 1; SUP = 0
Same trial against an illustrative 5-point margin
With the same results and an illustrative margin of 5 points, the lower limit of minus 0.0642 falls below minus 0.05, so non-inferiority is not shown, although the interval contains zero and the trial shows no significant difference. The margin is illustrative and lies below the article's M1 of 0.22.
p_T = 0.84; p_C = 0.86; n_T = 500; n_C = 500; z = 1.96; M = 0.05; d = -0.02; SE_d = 0.0226; L_d = -0.0642; U_d = 0.0242; NI = 0; SUP = 0
Illustrative larger trial that is non-inferior and statistically worse
With the article's cure proportions but 5,000 patients per arm, an illustrative size, the standard error falls to 0.0071 and the interval is minus 0.0340 at its lower end and minus 0.0060 at its upper end. Non-inferiority at the 10-point margin is shown, yet the whole interval lies below zero, so T is also statistically worse than C, the case the FDA guidance shows as result 6 of its Figure 2.
p_T = 0.84; p_C = 0.86; n_T = 5000; n_C = 5000; z = 1.96; M = 0.10; d = -0.02; SE_d = 0.0071; L_d = -0.034; U_d = -0.006; NI = 1; SUP = 0
Illustrative test treatment shown non-inferior and superior
If T instead cured 90% against 84% on C, with 500 patients per arm, the difference is 0.06 with standard error 0.0212 and the interval is 0.0185 at its lower end and 0.1015 at its upper end. The lower limit lies above minus 0.10 and above zero, so both readings are shown, and CPMP/EWP/482/99 section IV.1 allows a non-inferiority trial to be read as showing superiority in this case. The cure proportions are illustrative.
p_T = 0.90; p_C = 0.84; n_T = 500; n_C = 500; z = 1.96; M = 0.10; d = 0.06; SE_d = 0.0212; L_d = 0.0185; U_d = 0.1015; NI = 1; SUP = 1
Common errors
Concluding non-inferiority from a non-significant difference
ICH E9 section 3.3.2 calls it inappropriate to conclude non-inferiority from a non-significant test of no difference. In the second worked example the interval of minus 0.0642 to plus 0.0242 contains zero, so the difference is not significant, yet with a 5-point margin the lower limit lies 1.42 points beyond minus 0.05 and non-inferiority is not shown.
Reading non-inferior as no worse than the control
NI equal to 1 rules out a loss larger than M and nothing more. In the third worked example the 10-point margin is met while the whole interval lies below zero, so T is statistically worse than C by between 0.6 and 3.4 points, a result the FDA guidance section III.A (Figure 2, result 6) and Schumi and Wittes (2011) both describe as non-inferior and inferior at once. A model should carry d and SE_d, not a zero difference.
Reading an adverse outcome in the favourable outcome orientation
If the outcome is failure, with failure in 16% on T and 14% on C from 500 per arm, and d is entered as test minus control (plus 0.02), the lower limit is minus 0.0242 and a 5-point margin appears to be met. Entered in the orientation of HE-AS-ACTR-002, control minus test (minus 0.02), the lower limit is minus 0.0642 and non-inferiority is not shown (illustrative, derived). The reversed sign turns a failed non-inferiority result into a pass whenever M is above 0.024246 and at most 0.064246, the two unrounded lower limits (derived).
Using a 90% interval for a clinical non-inferiority trial
Two-sided 90% intervals are the norm for bioequivalence (CPMP/EWP/482/99 section II.4; HE-FM-ANDA-001), not for clinical non-inferiority, where the two-sided interval is 95%. With z of 1.645 the first worked example gives a lower limit of minus 0.0571 instead of minus 0.0642, so an illustrative 6-point margin appears to be met when the 95% interval does not meet it (derived). The 90% interval corresponds to a one-sided level of 0.05, twice the 0.025 the FDA guidance ties to the 95% interval.
Sources
ICH E9 on the lower margin and one-sided test for non-inferiority
International Conference on Harmonisation. ICH E9 Statistical principles for clinical trials. Step 4 guideline, 5 February 1998. Section 3.3.2 (Trials to Show Equivalence or Non-inferiority) states that only the lower margin is needed for the active control non-inferiority trial, that a one-sided interval should be used, that the interval approach has a one-sided test counterpart for the treatment difference (investigational product minus control) against the lower margin, and that concluding non-inferiority from a non-significant test of no difference is inappropriate. Section 5.5 (Estimation, Confidence Intervals and Hypothesis Testing) prefers one-sided type I errors at half the conventional two-sided level in regulatory settings.
CPMP points to consider on reading superiority and non-inferiority intervals
Committee for Proprietary Medicinal Products. Points to consider on switching between superiority and non-inferiority (CPMP/EWP/482/99). London, EMEA; 27 July 2000. Section II.1 treats a two-sided 95% interval for the difference that excludes zero as usually equivalent to a significant difference at the 5% level and takes values to the right of zero as better on the new treatment. Section II.3 states that for non-inferiority the two-sided 95% interval should lie entirely to the right of minus the margin. Section II.4 assumes two-sided 95% intervals for all clinical trials, one-sided intervals at 97.5% and 90% intervals for bioequivalence. Section III requires the margin to be set down in the protocol, and section IV.1 accepts superiority when the interval also lies above zero.
FDA guidance on the non-inferiority hypothesis and the 95%-95% method
US Food and Drug Administration. Non-inferiority clinical trials to establish effectiveness. Guidance for industry. Silver Spring, MD, FDA; November 2016. Section III.A (The Non-Inferiority Hypothesis) states the null hypothesis C minus T at least M and establishes non-inferiority when the upper bound of the two-sided interval for C minus T is less than M; its Figure 2, result 6, shows a study that meets M1 while T is inferior to C. Section III.C states that the 95% two-sided interval corresponds to a one-sided test size of 0.025. Section IV.C.1 (The Fixed Margin Approach) defines the 95%-95% method, with M1 the lower bound of the historical interval and the trial interval used to rule out C minus T above M2, and notes 10 to 15% margins on the risk difference scale for antibiotics.
ICH E10 on the upper limit for a non-inferiority margin
International Conference on Harmonisation. ICH E10 Choice of control group and related issues in clinical trials. Step 4 guideline, 20 July 2000. Section 1.5.1.1 (Historical Evidence of Sensitivity to Drugs Effects and Choosing the Non-inferiority Margin) states that the margin cannot be greater than the smallest effect size the active drug would reliably be expected to have compared with placebo in the setting of the planned trial. Section 1.5 (Assay Sensitivity) states that a successful non-inferiority trial generally does not contain direct evidence of assay sensitivity.
EMEA 2005 guideline on justifying the non-inferiority margin
Committee for Medicinal Products for Human Use. Guideline on the choice of the non-inferiority margin (EMEA/CPMP/EWP/2158/99). London, EMEA; adopted July 2005, in effect January 2006. Section 2 (General Considerations) asks for the choice of margin to be justified in the study protocol and states that defining the margin as a proportion of the difference between active comparator and placebo is not appropriate. Section 3 (Demonstrating Efficacy) asks for a conservative approach to the margin when constancy of effect cannot be assured.
Schumi and Wittes on the lower bound and the non-inferior yet inferior result
Schumi J, Wittes JT. Through the looking glass: understanding non-inferiority. Trials. 2011;12:106. The section Superiority, equivalence, and non-inferiority states that non-inferiority is judged on the lower bound of the interval for the difference against minus the margin, and that a narrow interval can show a new treatment non-inferior and inferior at once. The section Choosing the margin, conceptually describes the 95-95 method, with M1 the lower limit of the interval for the control effect against placebo and M2 a smaller margin that preserves a set fraction of it.
Newcombe comparison of intervals for a difference of independent proportions
Newcombe RG. Interval estimation for the difference between independent proportions: comparison of eleven methods. Statistics in Medicine. 1998;17(8):873-890. The study evaluates unconditional interval methods for the difference between binomial proportions, finds computationally simpler methods prone to aberrations and poor coverage, and reports that the methods of Mee and of Miettinen and Nurminen and a method combining Wilson score intervals for the two proportions perform well.
Canonical Identity
Stable URI · Machine-readable · Resolvable · CC BY 4.0