Begg Test

Begg Test for Small-Study Effects in Meta-Analysis

The Begg test, proposed by Begg and Mazumdar in 1994, asks whether the smaller trials in a meta-analysis report systematically different results from the larger ones. In health technology assessment this matters because a pooled relative effect usually becomes a parameter in a decision model, so a distorted evidence base passes straight into estimated QALY gains and cost-effectiveness results. An illustrative set of ten trials behind a model's hazard ratio shows how the statistic works and how little power the test has when studies are few. The Egger test asks the same question by regression, and the Cochrane Handbook reports recommendations that limit when any test of funnel plot asymmetry should be used.

Small-study effects and the funnel plot

A funnel plot shows each study's effect estimate against its standard error. Without bias or heterogeneity, small imprecise studies scatter widely and large studies cluster near the pooled effect, giving a symmetrical inverted funnel. The Cochrane Handbook calls a tendency for smaller studies to estimate different effects from larger ones a small-study effect, and treats the funnel plot as a general display of such effects rather than a picture of publication bias alone.

The Handbook lists several possible causes of asymmetry. Small studies with unfavourable results may go unreported, small trials with design flaws tend to show larger effects, and early small trials may recruit higher-risk patients who benefit more. Odds ratios and standardised mean differences are naturally correlated with their standard errors, which can create artefactual asymmetry, and chance can do the same. Begg and Mazumdar described their test as a direct statistical analogue of the funnel graph, so it inherits all of these explanations. A significant result signals asymmetry, not its cause.

The adjusted rank correlation statistic

The test correlates the ranks of standardised effect estimates with the ranks of their variances. Standardisation removes the pooled effect from each estimate and divides by the standard deviation of that difference, so that without small-study effects the standardised values show no trend with study size. The first step is the inverse-variance pooled estimate, given below.

$$\hat{\theta}_{IV} = \frac{\sum_{j=1}^{K} \hat{\theta}_j / \hat{\sigma}_j^2}{\sum_{j=1}^{K} 1 / \hat{\sigma}_j^2}$$

where $\hat{\theta}_{IV}$ is the fixed-effect pooled estimate, $\hat{\theta}_j$ is the effect estimate from study $j$ on an approximately normal scale (such as a log hazard ratio), $\hat{\sigma}_j^2$ is its within-study variance and $K$ is the number of studies. Each estimate is then standardised as follows.

$$t_j^{*} = \frac{\hat{\theta}_j - \hat{\theta}_{IV}}{\sqrt{v_j^{*}}}, \qquad v_j^{*} = \hat{\sigma}_j^2 - \left( \sum_{k=1}^{K} \hat{\sigma}_k^{-2} \right)^{-1}$$

where $t_j^{*}$ is the standardised effect of study $j$ and $v_j^{*}$ is the variance of the difference between that study's estimate and the pooled estimate, which allows for each study's own contribution to the pooled value.

The test is Kendall's rank correlation between the $t_j^{*}$ values and the variances $\hat{\sigma}_j^2$. A pair of studies is concordant when the study with the larger variance also has the larger standardised effect, and discordant when it has the smaller one. The score, correlation and test statistic are defined below.

$$S = C - D, \qquad \tau = \frac{S}{K(K-1)/2}, \qquad \operatorname{Var}(S) = \frac{K(K-1)(2K+5)}{18}, \qquad Z = \frac{S}{\sqrt{\operatorname{Var}(S)}}$$

where $S$ is Kendall's score, $C$ and $D$ are the numbers of concordant and discordant pairs, $K(K-1)/2$ is the number of pairs, $\tau$ is the rank correlation, $\operatorname{Var}(S)$ is the null variance of $S$ when there are no ties and $Z$ is compared with the standard normal distribution. When benefit is negative, as with log hazard ratios, exaggerated benefit in small studies produces a negative tau. Software differs in detail. The Kendall routine that Stata's meta bias command refers to applies a continuity correction, using $\lvert S \rvert - 1$ in the numerator, while the ranktest function in the R package metafor reports an exact p-value where it can. With few studies these choices can move a p-value across a conventional threshold, so a report should say which version it used.

Worked example with ten trials behind a model's hazard ratio

A cost-effectiveness model of a new treatment for a progressive disease needs a hazard ratio for progression against standard care. The ten trials below, and every number derived from them, are illustrative. Effects are log hazard ratios, so negative values favour the new treatment, and trials are listed from most to least precise.

1. Pool the estimates. The weights (one over the variance) sum to 743.66 and the weighted log hazard ratios sum to −129.00, so the pooled estimate is as follows.

$$\hat{\theta}_{IV} = \frac{-129.00}{743.66} = -0.1735$$

where $\hat{\theta}_{IV}$ is the pooled log hazard ratio, a hazard ratio of 0.841. The correction $c = 1/743.66 = 0.001345$ is subtracted from each variance to give $v_j^{*}$.

2. Standardise each trial. For trial 1 the calculation is as follows.

$$t_1^{*} = \frac{-0.12 - (-0.1735)}{\sqrt{0.0036 - 0.001345}} = \frac{0.0535}{0.0475} = 1.13$$

where $t_1^{*}$ is the standardised effect of trial 1; the other rows follow in the same way.

3. Count pairs. As the trials are in order of increasing variance, each is compared with every trial below it. Trial 1 has a higher $t^{*}$ than all nine later trials, giving 9 discordant pairs. Trial 2 is exceeded by trials 3, 4 and 6 (3 concordant) and lies above trials 5, 7, 8, 9 and 10 (5 discordant). Working down the table gives concordant counts of 0, 3, 0, 1, 3, 0, 2, 1, 1 and 0, and discordant counts of 9, 5, 7, 5, 2, 4, 1, 1, 0 and 0, so $C = 11$ and $D = 34$ out of 45 pairs.

4. Compute the statistic.

$$S = 11 - 34 = -23, \qquad \tau = \frac{-23}{45} = -0.511, \qquad \operatorname{Var}(S) = \frac{10 \times 9 \times 25}{18} = 125, \qquad Z = \frac{-23}{11.18} = -2.06$$

where $S$ is Kendall's score, $\tau$ the rank correlation, $\operatorname{Var}(S)$ the null variance for ten studies and $Z$ the test statistic, with a two-sided normal p-value of about 0.040. With the continuity correction $\lvert Z \rvert = 22/11.18 = 1.97$, a p-value of about 0.049, and the exact permutation p-value is about 0.047.

5. Interpret. A tau of −0.51 is a strong association, in which the less precise trials report larger benefits. Yet the p-value sits just under 0.05 and moves with the method of calculation, which shows how little room the test has even with ten trials, and it says nothing about why the small trials differ. A natural sensitivity analysis pools only the four largest trials, whose weights sum to 603.47 and whose weighted log hazard ratios sum to −85.08, giving the estimate below.

$$\hat{\theta}_{1\text{-}4} = \frac{-85.08}{603.47} = -0.1410$$

where $\hat{\theta}_{1\text{-}4}$ is the pooled log hazard ratio from trials 1 to 4, a hazard ratio of 0.868. The relative reduction in the hazard of progression falls from about 15.9% to about 13.2%. If the model's incremental QALYs scale roughly with that reduction and incremental costs are unchanged, the estimated benefit is about one sixth smaller under the large-trial estimate, and the ICER correspondingly higher (about 21%).

Why small-study effects matter for decision model inputs

Economic evaluations in HTA usually apply pooled relative effects to baseline risks inside a health economic model. The NICE manual (PMG36) states that relative treatment effects from randomised trials may be applied to baseline risk data for the population of interest, and it expects a systematic review done to a pre-defined protocol, with potential biases identified. A small-study effect is one such bias, and it enters the model through the treatment effect parameter.

If the pooled effect is exaggerated, the model overstates health gain in every cycle, and for a technology that costs more than its comparator the ICER is typically understated. Probabilistic analysis does not correct this, because the distribution sampled for the treatment effect is centred on the biased estimate. Parameter uncertainty and bias are different problems, and a narrow interval around a biased mean can make a value of information analysis look more settled than the evidence justifies. A positive asymmetry finding should therefore lead to scenario analyses in the model, such as pooling the larger trials only or using a Copas selection model, rather than a single adjusted figure presented as fact.

Low power and what a non-significant result means

The test's main weakness is statistical power. In their own simulations, Begg and Mazumdar found the test fairly powerful for meta-analyses of 75 studies but only moderately powerful with 25, and warned that bias cannot be ruled out when the test is not significant, while noting that where power was low there was often relatively little bias in the summary estimate.

A simulation study by Sterne, Gavaghan and Egger, modelled on 78 published meta-analyses and judging evidence of small-study effects at P < 0.1, made the gap concrete. With a control group event rate of 20% and no true treatment effect, moderate bias in meta-analyses of 10 trials was detected in 14.7% of simulations by the rank correlation test and 23.5% by the regression test, and severe bias in 31.1% and 56.1% respectively.

The arithmetic shows the same limit. The table gives the strongest possible result, every pair ordered the same way; with no ties, the exact two-sided p-value for a perfect ordering is $p = 2/K!$.

With four studies, perfect concordance cannot reach significance at 5% on the exact test, although the uncorrected normal approximation suggests it does. A non-significant Begg test in a small meta-analysis gives no assurance that a pooled effect is safe to use in a model.

How the Begg test compares with the Egger test

Egger and colleagues proposed in 1997 a regression approach to the same question. Each study's standard normal deviate (its effect divided by its standard error) is regressed on its precision, and an intercept away from zero indicates funnel plot asymmetry. The two tests look for the same pattern but can disagree on the same data.

Stata's documentation for meta bias states that regression-based tests tend to perform better on type I error and describes the Begg test as no longer recommended in the literature, provided for completeness. The Cochrane Handbook adds that the original Egger test is not recommended for odds ratios or standardised mean differences, and points to the Harbord and Peters tests for odds ratios.

Recommendations reported in the Cochrane Handbook

The Cochrane Handbook (version 6.5, chapter 13) reports recommendations made by Sterne and colleagues in 2011 for tests of funnel plot asymmetry in general. The Handbook does not name the rank correlation test, so applying these recommendations to the Begg test is an inference from its being a test of that kind.

• As a rule of thumb, tests should be used only when a meta-analysis includes at least 10 studies, because power is low with fewer.
• Tests should not be used when the studies are of similar size, with similar standard errors.
• Results should be read alongside visual inspection of the funnel plot; a contour-enhanced funnel plot can help separate non-reporting bias from other causes.
• When a test indicates asymmetry, non-reporting bias is only one possible explanation, and the reasons should be examined.

Citing a 2022 study by Nguyen and colleagues, the Handbook reports that only 28% of a random sample of 300 systematic reviews indexed in 2020 included a meta-analysis with at least 10 studies. For many appraisals a formal test is therefore inappropriate, and the judgement about missing evidence rests on searches of trials registers and other unpublished sources, comparison of fixed-effect and random-effects estimates where there is between-study heterogeneity, and sensitivity analyses.

Common misreadings of the Begg test

One error is to treat a non-significant result as evidence that the pooled effect is unbiased, a reading Begg and Mazumdar themselves warned against. The opposite error is to equate a significant result with publication bias, when poorer-quality small trials, genuinely different patients in small trials or the link between some effect measures and their standard errors can all produce the same pattern, and each calls for a different response in the model.

Sources

• Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics. 1994;50(4):1088-1101.
• Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629-634.
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. Sections 3.3.4 and 4.6.16.
• Page MJ, Higgins JPT, Sterne JAC. Chapter 13: Assessing risk of bias due to missing evidence in a meta-analysis. In: Higgins JPT, Thomas J, Chandler J, et al, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024.
• StataCorp. Stata Meta-Analysis Reference Manual: meta bias. College Station, TX: StataCorp LLC; 2025.
• StataCorp. Stata Base Reference Manual: spearman (Spearman's and Kendall's correlations). College Station, TX: StataCorp LLC; 2025.
• Sterne JAC, Gavaghan D, Egger M. Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. Journal of Clinical Epidemiology. 2000;53(11):1119-1129.
• Sterne JAC, Sutton AJ, Ioannidis JPA, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343:d4002.
• Viechtbauer W. ranktest: Rank correlation test for funnel plot asymmetry. Documentation for the metafor package for R, version 5.3-0. Accessed 1 October 2026.