Response to “Trust, but verify” by Banaschewski et al.

Banaschewski et al. (2016) question the quality of our Cochrane review on methylphenidate for children and adolescents with ADHD (Storebø et al., 2015). Our response to the criticism is as follows:

We found no significant difference in a subgroup analysis comparing parallel and cross-over trials, and we noted considerable statistical heterogeneity between the two trial groups. Furthermore, we had only end-of-trial data from the cross-over trials, which means that they cannot be considered to be independent data (Higgins & Green, 2011). It is not recommended to pool end-of-trial data from cross-over trials with parallel group trials (Higgins & Green, 2011). Had we done so, we would have risked introducing a “unit-of-analysis error”.

Banaschewski et al. (2016) suggest that four trials should have been excluded from our analyses because they compared methylphenidate to active controls. This is not the case. The “active control” of the four trials was indeed a cointervention delivered to participants in both the methylphenidate and the control group. The trials therefore provide data on the “value added” by methylphenidate and were identified in our published protocol as eligible for inclusion (Storebø et al., 2015). Sensitivity analysis showed that excluding these trials would have had a negligible effect on our results.

They also suggested excluding another trial that included children aged 3 to 6 years of age. Yet this trial meets our inclusion criteria, and it would have been incorrect to exclude it. Again, sensitivity analysis showed that exclusion of the trial would have had a negligible effect on our results. Excluding all five trials would not have changed our conclusions.

Cohen’s effect sizes are not based on empirical evidence, but are crude “rules of thumb” and are not as useful to clinicians as a more scientific approach, such as assessing intervention effects relative to a minimal relevant difference. Variability of the minimal relevant difference is important, which is why we reported the 95 % confidence interval of the transformed mean value in our review (Storebø et al., 2015).

We did not overlook information from the Coghill 2007 trial but twice emailed the authors for additional information. They did not respond. The information required is not available in the published study.

It is correct that we assessed seven and not six cross-over trials as being of low risk for bias. Our thanks for spotting this error. Adding the seventh trial to the subgroup analysis on “teacher-rated ADHD symptoms-cross over trials” demonstrates a significant difference between trials with high compared to low risk of bias (SMD -0.96 [-1.09 to -0.82] compared to -0.64 [-0.91 to -0.38]; p = .04).

We assessed the “vested interest” domain to be “unclear” in 48 trials or “high risk” in 71 trials as they were funded by the pharmaceutical industry and/or the authors had affiliations to the industry. This gives a total of 119 trials with this domain at “high risk of bias”.

All trials, irrespective of any vested interest bias, were regarded as having a high risk of bias because of broken outcome assessor blinding given the easily recognizable, well-known adverse effects of methylphenidate.

Therefore, no trial, including those funded by the pharmaceutical industry, showed a low risk of bias in all domains.

Comparing teacher-rated ADHD symptoms in the 14 trials rated as having a high risk of vested interest compared to the 5 trials rated as having a low risk of vested interest shows a significant difference even within this small sample (SMD -0.86 [-0.99 to -0.72] compared to -0.50 [-0.69 to -0.31]; p = .003).

Conclusions

The trial selection in our review is not flawed. The effect sizes are not too small. We followed a sound methodology for the assessment of bias, and our conclusions are not misleading.

References