Bias in meta-analysis detected by a simple, graphical test

Journal Article

Matthias Egger•1997

George Davey SmithMartin Schneider

Top 1% · 99th Percentile

57,277 citations · Statistics, Probability and Uncertainty

Open Access

TLDR

A simple graph can help spot bias in research reviews, but its results should be used carefully, especially when only a few small studies are included.

Summary

AI Generated

This content was automatically synthesized by Credo's AI models directly from the original source text.

AI summaries can make mistakes — double-check important details against the original source.

1 Study Aim

The main goal of this paper is to find out if a simple test using funnel plots (graphs that show study results against study size) can predict when the results of a meta-analysis (a study that combines results from several smaller studies) do not match the results of a single large trial. The authors also want to see how common bias is in published meta-analyses by checking for funnel plot asymmetry (an uneven shape in the graph that can signal bias). The study wants to see if a simple graph can show when research reviews are likely to be biased.

2 Study Design

The researchers searched Medline to find pairs of studies: one meta-analysis and one large trial on the same topic. They considered the results to agree if both pointed in the same direction and the meta-analysis estimate was within 30% of the large trial's result. They also examined funnel plots from 37 meta-analyses published in top medical journals between 1993 and 1996, and 38 meta-analyses from the Cochrane Database of Systematic Reviews. The main measure was how uneven the funnel plot was, using a regression method that checks if the graph is symmetrical. The study compared results from research reviews and big trials, and checked many published reviews for signs of bias using a simple graph.

3 Findings

The study reveals that, among eight pairs of meta-analyses and large trials, four pairs agreed and four did not. In every case where they disagreed, the meta-analysis showed a bigger effect than the large trial. Funnel plot asymmetry, which signals possible bias, was found in three out of four of these mismatched pairs, but in none of the pairs that agreed. When looking at published meta-analyses, 38% of those from leading journals and 13% from the Cochrane reviews showed signs of bias using the funnel plot test. The authors recommend that checking for bias with funnel plots should become a routine part of reviewing research, but they caution that this method is less reliable when only a few small studies are included in the analysis. The research found that a simple graph can often spot bias in research reviews, but it works best when there are enough studies included.

Abstract

Abstract Objective: Funnel plots (plots of effect estimates against sample size) may be useful to detect bias in meta-analyses that were later contradicted by large trials. We examined whether a simple test of asymmetry of funnel plots predicts discordance of results when meta-analyses are compared to large trials, and we assessed the prevalence of bias in published meta-analyses. Design: Medline search to identify pairs consisting of a meta-analysis and a single large trial (concordance of results was assumed if effects were in the same direction and the meta-analytic estimate was within 30% of the trial); analysis of funnel plots from 37 meta-analyses identified from a hand search of four leading general medicine journals 1993-6 and 38 meta-analyses from the second 1996 issue of the Cochrane Database of Systematic Reviews . Main outcome measure: Degree of funnel plot asymmetry as measured by the intercept from regression of standard normal deviates against precision. Results: In the eight pairs of meta-analysis and large trial that were identified (five from cardiovascular medicine, one from diabetic medicine, one from geriatric medicine, one from perinatal medicine) there were four concordant and four discordant pairs. In all cases discordance was due to meta-analyses showing larger effects. Funnel plot asymmetry was present in three out of four discordant pairs but in none of concordant pairs. In 14 (38%) journal meta-analyses and 5 (13%) Cochrane reviews, funnel plot asymmetry indicated that there was bias. Conclusions: A simple analysis of funnel plots provides a useful test for the likely presence of bias in meta-analyses, but as the capacity to detect bias will be limited when meta-analyses are based on a limited number of small trials the results from such analyses should be treated with considerable caution. Key messages Systematic reviews of randomised trials are the best strategy for appraising evidence; however, the findings of some meta-analyses were later contradicted by large trials Funnel plots, plots of the trials' effect estimates against sample size, are skewed and asymmetrical in the presence of publication bias and other biases Funnel plot asymmetry, measured by regression analysis, predicts discordance of results when meta-analyses are compared with single large trials Funnel plot asymmetry was found in 38% of meta-analyses published in leading general medicine journals and in 13% of reviews from the Cochrane Database of Systematic Reviews Critical examination of systematic reviews for publication and related biases should be considered a routine procedure

Primary Link Archive Pubmed DOI Openalex Full text (OA)