Sensitivity, specificity and misclassification of one candidate change-score cut-off against an anchor

Treats the change score as a diagnostic test and the anchor as the reference standard. At a candidate cut-off, sensitivity is the share of anchor-improved patients whose change reaches it and specificity the share of all other patients whose change falls below it; the ROC cut-off is the candidate with the smallest sum of the two misclassification proportions, the same as the largest Youden index.

Signature

Se_c = a_c / N_I; Sp_c = (N_N - b_c) / N_N; M_c = (1 - Se_c) + (1 - Sp_c); J_c = Se_c + Sp_c - 1
Inputs
InputsDefinitionUnit
a_cUnit: patients—
N_IUnit: patients—
N_NUnit: patients—
b_cUnit: patients—
Output
Se_cUnit: proportion—
Sp_cUnit: proportion—
M_cUnit: proportion, from 0 to 2—
J_cEqual to 1 minus M_cproportion

Function

Anchor-based thresholds for a minimal clinically important difference and their use for responder shares and trial size

Maps a meaningful step on an external anchor, usually a patient's global rating of change, onto the score scale of an outcome measure. The between-group estimate subtracts the mean change of patients reporting no change from that of patients reporting a small change; the ROC cut-off chooses the change score that best separates anchor-improved patients from the rest; the 95 per cent limit cut-off sits above almost all unchanged patients. A threshold then gives the share of patients reaching it in each arm, and a between-group difference sets the target difference of a sample size calculation. Distribution-based yardsticks are HE-FM-DMID-001 to HE-FM-DMID-003 on the Distribution-Based MID page. Notation follows the Minimal Clinically Important Difference article.

Computational function

  • Computational function: ROC threshold for a minimal important change from patients' change scores and anchor classification

    Computes sensitivity, specificity, the misclassification sum and the Youden index at every candidate cut-off from patients' change scores and their anchor classification, and returns the ROC threshold, the candidate with the smallest misclassification sum (the lowest such candidate when two tie). The inputs differ from the formula's: a change score and an improved flag for every patient and a list of candidate cut-offs, in place of the counts for one cut-off.

    Inputs and outputs: change: Each patient's change score; required. Unit: score points.; improved: Whether the anchor classes the patient as improved; required. Unit: logical.; cuts: Candidate cut-offs; default every observed change. Unit: score points.; sens, spec: Sensitivity and specificity at each cut-off (HE-FM-MCID-002). Unit: proportion.; misclass, youden: Misclassification sum and Youden index at each cut-off. Unit: proportion.; cutoff: ROC threshold. Unit: score points.

    Assumption: A patient reaches a cut-off when the change is at least the cut-off, and the anchor classification is fixed before any cut-off is tried; the result carries the bias noted by Terwee and colleagues when the share improved is far from one half.

    Worked example (Change scores matching the article's table): With 90 improved patients whose changes are 2 (10 patients), 4 (8), 6 (12), 8 (15) and 10 (45), and 110 others whose changes are 0 (80), 4 (14), 6 (8), 8 (5) and 10 (3), constructed here so that the counts reaching each cut-off match the article, the misclassification sums at 4, 6, 8 and 10 points are 0.384, 0.345, 0.406 and 0.527, and the ROC threshold is 6 points, as in the article. cuts = 4, 6, 8, 10; sens = 0.889, 0.800, 0.667, 0.500; spec = 0.727, 0.855, 0.927, 0.973; cutoff = 6

    Excel: With change scores in ChangeVec, improved flags (1 or 0) in ImprovedVec and a cut-off in CutVal, =SUMPRODUCT(--(ChangeVec>=CutVal),ImprovedVec)/SUM(ImprovedVec) returns sensitivity and =SUMPRODUCT(--(ChangeVec<CutVal),1-ImprovedVec)/(COUNT(ImprovedVec)-SUM(ImprovedVec)) specificity; copied over the cut-offs in CutVec they fill MisclassVec and YoudenVec, and =INDEX(CutVec,MATCH(MIN(MisclassVec),MisclassVec,0)) returns the threshold into ROCCut.

    R: roc_cut <- function(change, improved, cuts = sort(unique(change))) { se <- sapply(cuts, function(c) mean(change[improved] >= c)); sp <- sapply(cuts, function(c) mean(change[!improved] < c)); m <- (1-se)+(1-sp); list(table = data.frame(cut = cuts, sens = se, spec = sp, misclass = m, youden = se+sp-1), cutoff = cuts[which.min(m)]) } Base R only; roc_cut(c(rep(2, 10), rep(4, 8), rep(6, 12), rep(8, 15), rep(10, 45), rep(0, 80), rep(4, 14), rep(6, 8), rep(8, 5), rep(10, 3)), rep(c(TRUE, FALSE), c(90, 110)), c(4, 6, 8, 10)) returns the example.

    Python: def roc_cut(change, improved, cuts=None): cuts = cuts or sorted(set(change)); imp = [c for c, i in zip(change, improved) if i]; oth = [c for c, i in zip(change, improved) if not i]; rows = [{"cut": c, "sens": sum(x >= c for x in imp)/len(imp), "spec": sum(x < c for x in oth)/len(oth)} for c in cuts]; [r.update(misclass=2-r["sens"]-r["spec"], youden=r["sens"]+r["spec"]-1) for r in rows]; return {"table": rows, "cutoff": min(rows, key=lambda r: r["misclass"])["cut"]} Needs no imports; returns the same values as the R function.

    Test (ROC threshold has the largest Youden index): The Youden index at ROCCut equals the largest value in YoudenVec. Expected result: TRUE. FALSE shows MAX in place of MIN in the MATCH formula, which picks the 10-point cut-off with the largest misclassification sum. Excel check: =INDEX(YoudenVec,MATCH(ROCCut,CutVec,0))=MAX(YoudenVec)

    Common error (Counting only the about-the-same group as the reference for the ROC): The ROC analysis in the article compares the 90 improved patients with all 110 others; restricting the reference to the 70 rating themselves about the same, as the 95 per cent limit does, changes specificity at every cut-off and can move the threshold.

    Source: de Vet HC, Ostelo RW, Terwee CB, van der Roer N, Knol DL, Beckerman H, Boers M, Bouter LM. Minimally important change determined by a visual method integrating an anchor-based and a distribution-based approach. Quality of Life Research. 2007;16(1):131-142. doi:10.1007/s11136-006-9109-9 (full text read). Methods; Terwee CB, Peipert JD, Chapman R, Lai JS, Terluin B, Cella D, Griffiths P, Mokkink LB. Minimal important change (MIC): a conceptual clarification and systematic review of MIC estimates of PROMIS measures. Quality of Life Research. 2021;30(10):2729-2754. doi:10.1007/s11136-021-02925-y (full text read). Part 2.

    Se_c = sum_i [I(change_i >= c) * improved_i] / sum_i [improved_i]; Sp_c = sum_i [I(change_i < c) * (1 - improved_i)] / sum_i [1 - improved_i]; M_c = (1 - Se_c) + (1 - Sp_c); cutoff = argmin_c M_c

Try this function

Implementations

  • Excel

    Sensitivity, specificity, misclassification sum and Youden index of one cut-off from named cells

    With ImpReach, ImpTotal, NotImpReach and NotImpTotal named, the formulas return the sensitivity, the specificity, the misclassification sum and the Youden index, held in SensC, SpecC, MisclassSum and YoudenJ.

    =ImpReach/ImpTotal; =(NotImpTotal-NotImpReach)/NotImpTotal; =(1-SensC)+(1-SpecC); =SensC+SpecC-1

Assumptions

  • Anchor as reference standard with a fixed split of the cohort

    The anchor divides the cohort into improved and not improved before any cut-off is tried, and a patient reaches a cut-off when the change is at least the cut-off; in the article's example the 90 patients rating themselves better are improved and the 110 others are the reference group.

  • Share of improved patients near one half

    Terwee and colleagues note that the ROC estimate is biased when the share of improved patients is not 50 per cent; in the article's example it is 45 per cent, so a predictive modelling estimate would be a useful check.

Worked examples

  • Cut-off of 6 points in the article's cohort

    Of 90 improved patients 72 reach 6 points (sensitivity 0.800) and of 110 others 16 do (specificity 94 / 110 = 0.855); the misclassification sum of 0.345 is the smallest of the four cut-offs, so the ROC threshold is 6 points, as in the article.

    a_c = 72; N_I = 90; b_c = 16; N_N = 110; Se_c = 0.8; Sp_c = 0.855; M_c = 0.345; J_c = 0.655

  • Cut-off of 4 points in the article's cohort

    Eighty improved and 30 other patients reach 4 points, giving sensitivity 0.889, specificity 0.727 and a misclassification sum of 0.384, as in the article.

    a_c = 80; N_I = 90; b_c = 30; N_N = 110; Se_c = 0.889; Sp_c = 0.727; M_c = 0.384; J_c = 0.616

  • Cut-off of 10 points in the article's cohort

    Forty-five improved and 3 other patients reach 10 points, giving sensitivity 0.500, specificity 0.973 and a misclassification sum of 0.527, as in the article.

    a_c = 45; N_I = 90; b_c = 3; N_N = 110; Se_c = 0.5; Sp_c = 0.973; M_c = 0.527; J_c = 0.473

Common errors

  • Reporting the mean change of the minimally improved group as if it were the ROC estimate

    Terwee and colleagues note that in theory the mean change method overestimates the minimal important change; in the article's example it gives 8.0 points against the ROC threshold of 6, so a report should name its estimator.

  • Comparing ROC thresholds from cohorts with very different shares improved

    The ROC estimate is biased when the share of improved patients is not 50 per cent, so thresholds from a cohort where most patients improved and one where few did are not directly comparable; Terwee and colleagues recommend predictive modelling, which can be corrected for this bias.

Sources

  • De Vet and colleagues on the ROC cut-off for minimally important change

    de Vet HC, Ostelo RW, Terwee CB, van der Roer N, Knol DL, Beckerman H, Boers M, Bouter LM. Minimally important change determined by a visual method integrating an anchor-based and a distribution-based approach. Quality of Life Research. 2007;16(1):131-142. doi:10.1007/s11136-006-9109-9 (full text read). Methods: the change score is treated as a diagnostic test with the anchor as gold standard; sensitivity is the proportion of importantly improved or deteriorated persons correctly identified and specificity the proportion of not importantly changed persons correctly identified; the ROC cut-off point is the value for which ([1 minus sensitivity] + [1 minus specificity]) is smallest.

    View source →

  • Terwee and colleagues on the bias of ROC and mean change estimates

    Terwee CB, Peipert JD, Chapman R, Lai JS, Terluin B, Cella D, Griffiths P, Mokkink LB. Minimal important change (MIC): a conceptual clarification and systematic review of MIC estimates of PROMIS measures. Quality of Life Research. 2021;30(10):2729-2754. doi:10.1007/s11136-021-02925-y (full text read). Part 2 and Box 1: in theory MIC mean values overestimate the MIC; the MIC ROC will be biased if the percentage of improved patients is not 50%; the predictive modelling estimate is more precise, can be corrected for bias and is recommended as the best option.

    View source →

Canonical Identity

Persistent URI
https://healtheconomics.wiki/formula/sensitivity-specificity-and-misclassification-of-one-candidate-change-score-cut-off-against-an-anchor

Stable URI · Machine-readable · Resolvable · CC BY 4.0