Check and Double Check – the Cochrane review by Storebo et al. (2015) is indeed flawed
Storebo and colleagues have responded to our critizism (Banaschewski et al., 2016) in regard to the methodology and interpretation of their previous Cochrane analysis (Storebo et al., 2015). While we appreciate their effort to respond to our questions, we doubt that our points have been adequately addressed.
We have elaborated that 4 studies should be excluded from the analysis of the effect size of methylphenidate (MPH) due to “active control conditions” rather than placebo. Unfortunately, the authors consider those as mere “co-treatments” and refer to the “value added” by methylphenidate. Considering the effect of clonidine (0.58–0.61) (e. g. Bloch et al., 2009), parent training and behavioural therapy (Cohen’s d: 0.30–0.69) (Chan et al., 2016) on ADHD symptoms, it is obvious that the “added” effect size of MPH is significantly reduced compared to studies testing MPH against placebo. We would therefore like to repeat our notion that such an approach is substantially flawed as well as the underlying protocol. The same applies to the inclusion of preschool children who are at an age where off-label treatment with MPH is known to be less effective compared to older children (0.4–0.8) (Greenhill et al., 2006).
While the authors claim that sensitivity analyses reveal a negligible effect on the results, we have already demonstrated that excluding those 5 studies resulted in a large effect size of MPH (0.89) compared to the medium effect size (0.77), which was calculated by the authors. We think that this difference cannot be regarded as “negligible” at all.
The latter effect size has been the basis for the calculation of the “minimal clinical difference”. Unfortunately, the authors have not responded to our point, that the comparison to the minimal clinical difference may only be useful for assessing individual responses. Instead, they have described “effect sizes” as a “crude rule of thumb”, thus neutralizing their own data basis for the calculation of the minimal clinical difference. For this reason, and the apparent absence of a valid response, we are again faced with the problem that the clinical interpretation of the Cochrane has to be considered as flawed and even inadmissible.
Furthermore, the authors claim that the cross-over studies with end-point data have not been included in order to avoid a “unit-of-analysis-error”. However, they do not provide evidence for a significant difference of those studies compared to parallel-group studies. We think, this makes the argument more than questionable – especially in the light of the by far less stringent inclusion of the above mentioned studies.
The authors claim that typical adverse events of MPH, such as the loss of appetite and disturbed sleep, can easily be detected by teachers. However, it is a fact that teachers can neither control the amount of food that children eat in the schoolyard, nor assess their sleep quality at night. We thus believe that the assumed “broken outcome assessor blinding” and the derived bias assessment should be refuted. It is the repetition of a wrong argument as shown by criticism (e. g. Hoekstra and Buitelaar, 2016). If the authors are not able to provide evidence for their claim, we would ask them to abstain from it – or at least handle criticizm more respectful by not ignoring it.
There is another point we would like to make about the – as we think – inappropriate generalization of bias assessment in regard to the funding of industrial companies, which has been omitted by the authors. Indeed, we have provided some examples of industrial funding of producers of alternative medication, thus implying a bias even contrary to the effectiveness of MPH.
Storebo et al. (2015) criticised the studies Coghill et al., 2007 and Coghill et al., 2013 for providing insufficient information. While this is true for Coghill et al., 2007, it is not for Coghill et al., 2013, which provided the information on data handling in a table. We have reconfirmed that the authors have responded to the first request by Storebo et al. in regard to Coghill et al., 2007. In regard to Coghill et al., 2013 the authors had not been contacted.
Finally, the authors compared the effect size of those 5 studies with low vested bias to those 14 with high or unclear vested bias, revealing lower effect sizes in the low bias group. Unfortunately, the authors have failed to mention that we have identified 3 out of the 5 studies, classified as “low vested bias”, as inapt for inclusion (Brown et al., 1985; Firestone et al., 1981; Jensen et al. 1999). The discrepant effect size is thus not based on the vested interest domain, but rather on the questionable study inclusion procedure of Storebo and colleagues.
Interestingly, the authors have – to some extent – realized that the MTA study (Jensen et al., 1999) should not have entered the analysis due to further study details, e. g.:
“We have discussed whether to include the MTA study, as not all participants randomly assigned to medication (combined treatment and medication management group) received methylphenidate.[…] However, we have chosen to use the data from MTA, as it is such a large and well-known study.” (p 272/273; Storebo et al., 2015)
We are unaware of a Cochrane guideline, defining special inclusion criteria for studies based on their degree of publicity.
Let us conclude that the Cochrane review by Storebo et al. (2015) is indeed flawed in terms of study selection and risk of bias assessment. We think that the clinical interpretation of the flawed analysis should not be admissible and lacks careful consideration.
References
Literature
(2016) Trust, but verify. The errors and misinterpretations in the Cochrane analysis by O. J. Storebo and colleagues on the efficacy and safety of methylphenidate for the treatment of children and adolescents with ADHD. Zeitschrift für Kinder und Jugendpsychiatrie und Psychotherapie. doi: 10.1024/1422-4917/a000433
(1985). Methylphenidate and cognitive therapy: A comparison of treatment approaches with hyperactive boys. Journal of Abnormal Child Psychology, 13, 69–87.
(2009). Meta-analysis: treatment of attention-deficit/hyperactivity disorder in children with comorbid tic disorders. Journal of the American Academy of Child and Adolescent Psychiatry, 48(9), 884–893.
(2016). Treatment of Attention-Deficit/Hyperactivity Disorder in Adolescents: A Systematic Review. JAMA: the Journal of the American Medical Association, 315(18), 1997–2008. http://doi.org/10.1001/jama.2016.5453.
(2013) European, randomized, phase 3 study of lisdexamfetamine dimesylate in children and adolescents with attention-deficit/hyperactivity disorder. European Neuropsychopharmacology 23(10), .
(2007) The neuropsychological effects of chronic methylphenidate on drug-naive boys with attention-deficit/hyperactivity disorder. Biological Psychiatry 62(9): 954–62.
(1981) Differential effects of parent training and stimulant medication with hyperactives: a progress report. Journal of the American Academy of Child and Adolescent Psychiatry, 20, 135–147.
, (2006). Efficacy and safety of immediate-release methylphenidate treatment for pre-schoolers with ADHD. Journal of the American Academy of Child and Adolescent Psychiatry, 45(11), 1284–1293.
(2016). Response to: The evidence base of methylphenidate for children and adolescents with attention-deficit/hyperactivity disorder is in fact flawed. European Child and Adolescent Psychiatry. http://doi.org/10.1007/s00787-016-0856-z.
, (1999). A 14-month randomized clinical trial of treatment strategies for attention-deficit/hyperactivity disorder. Archives of General Psychiatry, 56, 1073–1086.
, (2015). Methylphenidate for children and adolescents with attention deficit hyperactivity disorder (ADHD). Cochrane Database of Systematic Reviews, Issue 11. Art. No.: CD009885. DOI: 10.1002/14651858.CD009885.pub2.
