The between-trial variance for this model was very small, indicating that most between-study variability was explained by these four variables.
The mean squared error was especially large when the between-study variability was large (table 1).
However, the type I error rate of these tests substantially exceeded 0.05, especially when a large between-study variability was present.
Yet in spite of some differences in the respective endpoints, between-study variability was quite low among all the eight trials which were included (I2 = 25%).
When statistically significant positive results were 10 times more likely to be included and the same amount of between-study variability was present, the random-effects meta-analysis overestimated the mean log-OR by 0.83 on average.
A higher probability of including statistically significant positive outcomes led to a drastic increase of the bias and the mean squared error, especially when a large between-study variability was present (table 1).
To test statistical heterogeneity across the study estimates, we estimated the variance corresponding to the between-study variability, and tested the null hypothesis that the between-study variability was absent.
For example, when statistically significant positive results were four times more likely to be included and a large between-study variability was present, more than 90% of the meta-analyses of 50 and 100 studies wrongly concluded that the mean effect size differed from zero.
A visual examination of the funnel plots indicated that a one-sided selection process based on the statistical significance introduced little asymmetry to the funnel plot both when the between-study variability was small (compare figure 2A, C) and large (compare figure 2B, D).
When statistically significant positive results were four times more likely to be included than other outcomes and a large between-study variability was present, more than 90% of the meta-analyses of 50 and 100 studies wrongly showed that the mean effect size differed from zero.
One potential cause of between-study variability is the use of different models (i.e. generations) of the Actigraph.
