Expected number of unpublished studies implied by Copas selection parameters for two standard error groups

Copas and Shi translate a pair of selection parameters into the expected number of unpublished studies, M, the sum over the published studies of (1 minus P_i) / P_i, with P_i the publication probability of HE-FM-COPAS-001 at study i's standard error. The formula is written for two groups of published studies that share a standard error within each group, n_1 at s_1 and n_2 at s_2, as in the article's example of two large and ten small trials; each further standard error adds a term of the same form, which the R and Python code of HE-CF-COPAS-001 handles. Each published study with probability P stands in expectation for (1 minus P) / P unpublished studies of the same precision, so groups with a low publication probability dominate M; in the article's moderate scenario 18.57 of the 18.68 come from the ten small trials. The step that turns P into M is the same algebra as the bed turnover interval of HE-FM-BTO-004 in another setting; the Copas content is the chain from gamma_0, gamma_1 and the standard errors.

Signature

u_1 = gamma_0 + gamma_1 / s_1; u_2 = gamma_0 + gamma_1 / s_2; P_1 = Phi_1; P_2 = Phi_2; M = n_1 * (1 - P_1) / P_1 + n_2 * (1 - P_2) / P_2
Inputs
InputsDefinitionUnit
gamma_0Intercept of the Copas probit selection equation, fixed at one grid pointstandard normal scale
gamma_1Coefficient of the precision 1 / s in the Copas probit selection equation, at zero or above, fixed at one grid pointeffect scale units
s_1Standard error shared by the published studies of the first group, the large trials in the articleeffect scale units
s_2Standard error shared by the published studies of the second group, the small trials in the articleeffect scale units
Phi_1Standard normal distribution function at u_1, entered from NORM.S.DIST(u_1, TRUE) or pnorm(u_1)probability
Phi_2Standard normal distribution function at u_2, entered from NORM.S.DIST(u_2, TRUE) or pnorm(u_2)probability
n_1Number of published studies with standard error s_1studies
n_2Number of published studies with standard error s_2studies
Output
u_1Index gamma_0 + gamma_1 / s_1 of the probit selection equation for the first groupstandard normal scale
u_2Index gamma_0 + gamma_1 / s_2 of the probit selection equation for the second groupstandard normal scale
P_1Probability that a study with standard error s_1 is publishedprobability
P_2Probability that a study with standard error s_2 is publishedprobability
MExpected number of studies left unpublished, given the published ones and the assumed gamma_0 and gamma_1studies

Function

Copas selection model publication probability and implied number of unpublished studies

Takes the two selection parameters of the Copas selection model, gamma_0 and gamma_1, and the standard errors of the published studies, and returns each study's probability of publication, Phi(gamma_0 + gamma_1 / s), and the number of unpublished studies those parameters imply, M. In a Copas sensitivity analysis gamma_0 and gamma_1 are fixed over a grid, and at each point the mean effect theta, the heterogeneity tau and the correlation rho are estimated by maximising the likelihood of the published estimates conditional on publication. That adjusted estimate has no closed form and comes from software such as the copas function of the R package metasens, so it is not written as a formula here; the publication probability and M are the quantities that label each grid point. The unadjusted estimate the analysis starts from is the common-effect inverse-variance mean of HE-FM-ADMA-001 (random-effects form HE-FM-ADMA-005), and the article's unadjusted 0.3671 is HE-FM-ADMA-001 applied to its twelve illustrative trials. A rank test for small-study effects is HE-FN-BEGG-001, with HE-FM-BEGG-002 and HE-CF-BEGG-001. Carrying an adjusted odds ratio into a decision model uses HE-FM-NMA-004 for the response probability on the new therapy and HE-FM-ICER-001 for the cost per extra responder. Notation follows the Copas Model article.

Computational function

  • Computational function: Copas selection parameters and implied unpublished studies from two assumed publication probabilities

    Takes the publication probabilities assumed for studies at two standard errors, the way the article sets each scenario, solves the selection equation of HE-FM-COPAS-001 at both for gamma_1 and gamma_0, and returns them with the implied number of unpublished studies of HE-FM-COPAS-002 for the published studies at those standard errors. With quantiles z_a = Phi^-1(P_a) and z_b = Phi^-1(P_b), gamma_1 = (z_a minus z_b) / (1 / s_a minus 1 / s_b) and gamma_0 = z_b minus gamma_1 / s_b. Huang and colleagues set the selection parameters of their simulation the same way, by solving two publication probability equations for studies of 20 and 500 patients in a selection equation written on the square root of the sample size; the closed form above, for the equation in 1 / s, is derived. The copas function of metasens chooses its grid of gamma_0 and gamma_1 to cover a range of publication probabilities for the study with the largest standard error, and repeating the calculation over pairs of probabilities builds the selection columns of the article's scenario table. The adjusted effect at each pair needs the conditional maximum likelihood fit of the copas software and is not computed here.

    Inputs and outputs: P_a and P_b are the publication probabilities assumed for a study with standard error s_a and one with s_b, with s_a below s_b; z_a and z_b are their standard normal quantiles, entered from NORM.S.INV or qnorm; n_a and n_b are the numbers of published studies at each standard error. Outputs are gamma_1 and gamma_0, the selection parameters on the standard normal scale, and M, the implied number of unpublished studies. The R and Python versions take vectors of probability pairs and return one row per scenario.

    Assumption: P_a is at least P_b, so gamma_1 is zero or above (HE-AS-COPAS-002); both probabilities are above 0 and below 1, so the quantiles are finite; the two standard errors differ; published studies at each standard error share the probability (HE-AS-COPAS-005).

    Worked example (Article's moderate scenario): Probabilities of 0.95 for the large trials and 0.35 for the small ones give gamma_1 = 2.0302 / 7.5 = 0.2707 and gamma_0 = -1.062, as in the article's step 2, and 18.68 implied unpublished trials. P_a = 0.95; P_b = 0.35; z_a = 1.6449; z_b = -0.3853; s_a = 0.10; s_b = 0.40; n_a = 2; n_b = 10; gamma_1 = 0.2707; gamma_0 = -1.062; M = 18.68

    Worked example (Article's severe scenario): Probabilities of 0.90 and 0.15 give gamma_1 = 0.3091, gamma_0 = -1.8091 and 56.89 implied unpublished trials, the article's 56.9. P_a = 0.90; P_b = 0.15; z_a = 1.2816; z_b = -1.0364; s_a = 0.10; s_b = 0.40; n_a = 2; n_b = 10; gamma_1 = 0.3091; gamma_0 = -1.8091; M = 56.89

    Worked example (Equal probabilities): With both probabilities at 0.5 the quantiles are zero, so gamma_1 and gamma_0 are zero, publication does not depend on precision, and each of the twelve published trials stands for one unpublished trial, a limiting case. P_a = 0.5; P_b = 0.5; z_a = 0; z_b = 0; s_a = 0.10; s_b = 0.40; n_a = 2; n_b = 10; gamma_1 = 0; gamma_0 = 0; M = 12

    Excel: With the probabilities in cells named PubProbA and PubProbB, the standard errors in StdErrA and StdErrB and the counts in TrialsA and TrialsB, a cell named GammaOneCF holds =(NORM.S.INV(PubProbA)-NORM.S.INV(PubProbB))/(1/StdErrA-1/StdErrB), a cell named GammaZeroCF holds =NORM.S.INV(PubProbB)-GammaOneCF/StdErrB and a cell named ImpliedUnpubCF holds =TrialsA*(1-PubProbA)/PubProbA+TrialsB*(1-PubProbB)/PubProbB. For a grid of scenarios put one pair of probabilities per row and fill the same formulas down with relative references.

    R: copas_scenario <- function(Pa, Pb, sa, sb, na, nb) { g1 <- (qnorm(Pa)-qnorm(Pb))/(1/sa-1/sb); g0 <- qnorm(Pb)-g1/sb; M <- na*(1-Pa)/Pa+nb*(1-Pb)/Pb; data.frame(Pa = Pa, Pb = Pb, gamma1 = g1, gamma0 = g0, M = M) } Run copas_scenario(c(0.95, 0.90), c(0.35, 0.15), 0.10, 0.40, 2, 10) for the article's two selection scenarios; it returns gamma1 of 0.2707 and 0.3091, gamma0 of -1.0620 and -1.8091 and M of 18.677 and 56.889.

    Python: def copas_scenario(Pa, Pb, sa, sb, na, nb): from statistics import NormalDist; q = NormalDist().inv_cdf; return [(pa, pb, (q(pa)-q(pb))/(1/sa-1/sb), q(pb)-(q(pa)-q(pb))/(1/sa-1/sb)/sb, na*(1-pa)/pa+nb*(1-pb)/pb) for pa, pb in zip(Pa, Pb)] Standard library only; copas_scenario([0.95, 0.90], [0.35, 0.15], 0.10, 0.40, 2, 10) returns the same rows as R.

    Test (Selection parameters reproduce both assumed probabilities): NORM.S.DIST of the index built from GammaZeroCF and GammaOneCF at each standard error returns PubProbA and PubProbB, within 1E-9 in total. Expected result TRUE. FALSE shows gamma_1 doubled or halved, or divided by the difference in standard errors instead of the difference in precisions, which gives gamma_1 = -6.767 and a probability of almost 0 at s_a. Excel check =ABS(NORM.S.DIST(GammaZeroCF+GammaOneCF/StdErrA,TRUE)-PubProbA)+ABS(NORM.S.DIST(GammaZeroCF+GammaOneCF/StdErrB,TRUE)-PubProbB)<1E-9

    Test (Article's moderate scenario count): With PubProbA 0.95, PubProbB 0.35, TrialsA 2 and TrialsB 10, ImpliedUnpubCF holds 18.6767 within 0.001. Expected result TRUE. FALSE shows 1 / P in place of (1 minus P) / P, which returns 30.6767. Excel check =ABS(ImpliedUnpubCF-18.6767)<0.001

    Common error (Standard errors in place of precisions): Dividing the difference in quantiles by s_a minus s_b, which is -0.30, instead of 1 / s_a minus 1 / s_b, which is 7.5, gives gamma_1 = -6.767 in the moderate scenario, a selection equation in which the most precise studies are the least likely to be published (computed here for illustration).

    Source: Huang A, Komukai S, Friede T, Hattori S. Using clinical trial registries to inform Copas selection model for publication bias in meta-analysis. Research Synthesis Methods. 2021;12(5):658-673 (preprint arXiv:2005.14396 full text read; journal abstract read). Preprint section 3.1, on fixing the selection parameters for sensitivity analysis and translating them into the expected number of unpublished studies, and preprint section 4.2, Data generation, which derives (alpha_0, alpha_1) by solving P20 = Phi(alpha_0 + alpha_1 sqrt(20)) and P500 = Phi(alpha_0 + alpha_1 sqrt(500)). Carpenter J, Rücker G, Schwarzer G. copas: an R package for fitting the Copas selection model. The R Journal. 2009;1(2):31-36 (full text read). Equation (4), Pr(Z_i > 0) = Phi(gamma_0 + gamma_1 / s_i), which the two-point solution inverts. Schwarzer G, Carpenter JR, Rücker G. metasens R package version 1.5-3 reference manual, function copas, Details, on the grid of gamma0 and gamma1 values corresponding to a range of publication probabilities for the study with the largest standard error.

    gamma_1 = (z_a - z_b) / (1 / s_a - 1 / s_b); gamma_0 = z_b - gamma_1 / s_b; M = n_a * (1 - P_a) / P_a + n_b * (1 - P_b) / P_b

Try this function

Implementations

  • Excel

    Copas implied unpublished studies for two standard error groups in Excel

    With the selection parameters in cells named GammaZeroGrp and GammaOneGrp, the group standard errors in StdErrGroupA and StdErrGroupB and the published counts in TrialsGroupA and TrialsGroupB, the five formulas fill cells named IndexGroupA, IndexGroupB, PubProbGroupA, PubProbGroupB and ExpectedUnpublished.

    =GammaZeroGrp+GammaOneGrp/StdErrGroupA; =GammaZeroGrp+GammaOneGrp/StdErrGroupB; =NORM.S.DIST(IndexGroupA,TRUE); =NORM.S.DIST(IndexGroupB,TRUE); =TrialsGroupA*(1-PubProbGroupA)/PubProbGroupA+TrialsGroupB*(1-PubProbGroupB)/PubProbGroupB

Assumptions

  • Selection parameters fixed when counting implied unpublished studies

    M is a consequence of the assumed gamma_0 and gamma_1, not an estimate from the published studies. Huang and colleagues present it as a more interpretable translation of the two sensitivity parameters, and propose using information in clinical trial registries to estimate all the parameters of the model.

  • Studies in each Copas group share one standard error

    Every published study in a group has the same standard error and so the same publication probability, as in the article's example with two large trials at 0.10 and ten small trials at 0.40. With distinct standard errors each study forms its own group with a count of 1.

  • Unpublished studies per published study in the Copas count

    If studies of a given precision are published independently with probability P, the expected number left unpublished for each one published is (1 minus P) / P, the mean number of failures per success of a negative binomial count (derived). P must be above zero; as P falls towards zero, M grows without bound.

Worked examples

  • Implied unpublished trials in the article's moderate scenario

    With the moderate scenario's gamma_0 = -1.062 and gamma_1 = 0.2707, the two large trials (standard error 0.10) are published with probability 0.95 and the ten small trials (0.40) with probability 0.35. M is 0.105 + 18.571 = 18.68, the article's figure, so about 19 trials, nearly all small, would sit unpublished behind the twelve found (illustrative inputs; Phi values rounded to two decimals, as in the article).

    gamma_0 = -1.062; gamma_1 = 0.2707; s_1 = 0.10; s_2 = 0.40; n_1 = 2; n_2 = 10; u_1 = 1.645; u_2 = -0.38525; Phi_1 = 0.95; Phi_2 = 0.35; P_1 = 0.95; P_2 = 0.35; M = 18.68

  • Implied unpublished trials in the article's severe scenario

    The article's severe scenario, probabilities of 0.90 and 0.15, corresponds to gamma_1 = 0.3091 and gamma_0 = -1.8091 (HE-CF-COPAS-001). M is 0.222 + 56.667 = 56.89, about 57 unpublished trials, as in the article's scenario table (illustrative inputs; Phi values rounded to two decimals).

    gamma_0 = -1.8091; gamma_1 = 0.3091; s_1 = 0.10; s_2 = 0.40; n_1 = 2; n_2 = 10; u_1 = 1.2819; u_2 = -1.03635; Phi_1 = 0.90; Phi_2 = 0.15; P_1 = 0.90; P_2 = 0.15; M = 56.89

  • Implied unpublished trials with almost no selection

    With gamma_1 = 0 and gamma_0 = 3 every trial is published with probability 0.99865 whatever its size, and the twelve trials imply only 0.0162 unpublished ones. M reaches the zero of the article's no-selection row only in the limit as P reaches 1 (illustrative).

    gamma_0 = 3; gamma_1 = 0; s_1 = 0.10; s_2 = 0.40; n_1 = 2; n_2 = 10; u_1 = 3; u_2 = 3; Phi_1 = 0.99865; Phi_2 = 0.99865; P_1 = 0.99865; P_2 = 0.99865; M = 0.0162

Common errors

  • Average publication probability used for the Copas unpublished count

    Because (1 minus P) / P is convex in P, giving all twelve trials of the moderate scenario their average probability of 0.45 returns 14.67 unpublished trials instead of 18.68, about a fifth too few (computed here for illustration). The count has to be built group by group or study by study.

  • Total studies run reported as the Copas unpublished count

    n / P counts the published studies as well as the unpublished ones. In the moderate scenario about 30.68 trials would have been run, of which 18.68 are unpublished; reporting 30.68 adds the twelve published trials to the count.

  • One grid point's implied count taken as the number of missing studies

    M follows from the assumed selection, not from the data. In the article's illustration the moderate and severe scenarios imply about 19 and about 57 unpublished trials, yet with the correlation re-estimated both reproduce the group means, so the published trials cannot choose between them. The count is a figure to set against trial registries, which Huang and colleagues use to inform the model.

Sources

  • Huang and colleagues on the Copas and Shi count of unpublished studies

    Huang A, Komukai S, Friede T, Hattori S. Using clinical trial registries to inform Copas selection model for publication bias in meta-analysis. Research Synthesis Methods. 2021;12(5):658-673 (preprint arXiv:2005.14396 full text read; journal abstract read). Preprint section 3.1, Brief review of the sensitivity analysis method by Copas and Shi, states that (alpha_0, alpha_1) were fixed as sensitivity parameters and (theta, tau, rho) estimated by maximising the conditional log-likelihood, and that according to the formula by Copas and Shi one can translate (alpha_0, alpha_1) into the expected number of unpublished studies by M = sum over the N published studies of {1 minus P(Y_i > 0 | s_i)} / P(Y_i > 0 | s_i), which is more interpretable. Their alpha_0, alpha_1 and Y_i are the gamma_0, gamma_1 and Z_i of the article. The abstract proposes using information in clinical trial registries to estimate all unknown parameters.

    View source →

  • Carpenter, Rücker and Schwarzer on the approximate number of unpublished studies

    Carpenter J, Rücker G, Schwarzer G. copas: an R package for fitting the Copas selection model. The R Journal. 2009;1(2):31-36 (full text read). Figure 1, the summary.copas output for nine trials of phenobarbital and periventricular haemorrhage, lists for each probability of publishing the study with the largest standard error the approximate number of studies the model suggests remain unpublished (N.unpubl), 0 at a probability of 1.00 and 9 at 0.37. The paper does not print the formula behind N.unpubl.

    View source →

Canonical Identity

Persistent URI
https://healtheconomics.wiki/formula/expected-number-of-unpublished-studies-implied-by-copas-selection-parameters-for-two-standard-error-groups

Stable URI · Machine-readable · Resolvable · CC BY 4.0