SYSTEM Research note on: Recognizing statistical in-between-trials heterogeneity

CONTEXT: Quantitative systematic reviews need to consider clinical, methodological and statistical heterogeneity of trials or datasets from trials before combining their results in meta-analyses, as combining results from heterogeneous datasets and/or trials will lead to meaningless results and thus should be avoided. Potential statistical heterogeneity between trials/datasets that have been considered as being clinically/methodologically homogeneous is usually investigated using I2 – test and Cochrane Q statistics.

PROBLEM: However, both, I2 – test and Cochrane Q statistics are tests with low statistical power, which gives even weaker results the smaller the number of trials/datasets is (n < 15).

SUGGESTED SOLUTION: Therefore, statistical in-between-trial heterogeneity should be assessed using the point estimate together with its 95% Confidence interval of the I2 – test. Galbraith plots are to be used to identify outlying datasets/trials, which then should be further investigated for potential effect modifiers using regression analysis.