LTE Exchange in J Clin Psych Re Statistical Methods, Predicting Relapse After ECT
LETTER TO THE EDITOR
How to Reliably Predict Relapse After Electroconvulsive Therapy?
J Clin Psychiatry 2022;83(1):21lr14269
To cite: Lambrichts S, Vansteelandt K, Obbels J, et al. How to reliably predict relapse after electroconvulsive therapy? J Clin Psychiatry. 2022;83(1):21lr14269.
aKU Leuven, Department of Neurosciences, Research Group Psychiatry, Neuropsychiatry, Academic Center for ECT and Neuromodulation (AcCENT), University Psychiatric Center KU Leuven (UPC KU Leuven), Kortenberg, Belgium
*Corresponding author: Simon Lambrichts, MD, University Psychiatric Center KU Leuven (UPC KU Leuven), Leuvensesteenweg 517, 3070 Kortenberg, Belgium (simon.lambrichts@upckuleuven.be).
To the Editor: We thank Dr Andrade for his comment1 on our study of relapse after abrupt discontinuation of maintenance electroconvulsive therapy (M-ECT) during the COVID-19 pandemic.2 Dr Andrade declares his concerns about the protection against a type I error in our analyses examining the association between several clinical and treatment characteristics and relapse. For both models used in our study, the results of the test of the overall statistical significance (testing the null hypothesis that all regression coefficients are zero) were reported. This test protects against a type I error. As we reported two models, one could rightly argue that the Bonferroni procedure should have been used for both overall tests. For both models used in our study, the P value for the overall test (< .0001) was below the Bonferroni-corrected significance threshold (.05/2). Besides, both models should not be seen as “additive” evidence for the same hypothesis. We used the same model twice with “diagnosis” in the first model being substituted by “indication” in the second model because both seemed relevant from a clinical point of view, yet it was not possible to include both predictors in the same model as they were highly correlated.
Contrary to the impression of Dr Andrade, the reference category for categorical predictor variables was clearly stated in both models, as illustrated in Table 2 of our article.2 In this table, we reported all possible pairwise comparisons, each time indicating which categories were compared.
Dr Andrade raises two highly relevant issues in data analysis: overfitting and confounding. In our “naturalistic experiment,” we aimed to avoid using more predictor variables in our model than the sample size allowed for (ie, prevention of overfitting). At the same time, we intended to include important predictor variables (ie, prevention of confounding). In our search for an optimal balance, we decided to include all 5 predictor variables (based on their clinical relevance, or evidence base, or both), yielding an events per variable value of 7.2 (36 patients relapsed; 5 predictor variables), which is below the suggested 10–15. Dr Andrade correctly emphasizes that recruiting larger samples contributes to the solution of overfitting and confounding (partially). However, in our study, following the forced discontinuation of M-ECT, the sample size was determined by the exceptional circumstances. Nevertheless, we fully agree with Dr Andrade’s plea for the recruitment of larger samples. Clearly, recruiting large samples is a major challenge in the ECT research field in general, and in post-ECT follow-up studies in particular. This calls for multicenter collaborations for conducting randomized controlled trials and prospective cohort studies.
Altogether, improving long-term outcome following ECT in patients with severe mood and psychotic disorders—which is high priority for the field3—requires methodologically rigorously conducted studies. More specifically, further work is needed to develop prognostic models,4 as reliable prediction of individuals’ risk of relapse may enable more efficient allocation of interventions to prevent relapse, improve quality of life for patients, and reduce economic cost on society.
Published online: December 7, 2021.
Potential conflicts of interest: None.
Funding/support: None.
LETTER TO THE EDITOR
Interpreting Exploratory Analyses: Reply to Lambrichts et al
Chittaranjan Andrade, MDa,*
To the Editor: Dr Lambrichts et al1 examined an important subject: predictors of relapse after discontinuation of maintenance electroconvulsive therapy (M-ECT). Their study was special because it emerged out of a natural experiment: M-ECT was perforce abruptly discontinued in all patients because of service delivery constraints during the COVID pandemic. In their response2 to my commentary3 on their study,1 Lambrichts and colleagues state that the P values for the test of overall significance of their Cox regression models were both below the Bonferroni-corrected threshold for statistical significance. I do not contest this, but what if this was so because of overfitting, the risk of which they acknowledge2? This is food for thought. But my concern with false-positive findings was also more general. No primary hypothesis was stated in the study,1 and the statistical significance of many independent variables was tested in exploratory analyses in regressions that were run in parallel. Furthermore, as they clarify,2 in Table 2 of their paper1 they reported statistical test results of “all possible pairwise comparisons.” In other words, there were many exploratory statistical analyses conducted with no primary hypothesis stated.
It is well known that the larger the number of statistical tests conducted, the greater the likelihood that some P values will fall below the .05 threshold for declaration of statistical significance. In exploratory analyses, such statistically significant values do not necessarily represent findings that are true in the population; they could be chance findings, and hence false-positive results for hypotheses related to these findings when the hypotheses are stated after the findings are known.
This concern, of course, applies to all exploratory analyses, and so it is not an error; it is merely a limitation. Hypotheses generated in exploratory analyses can certainly be important but need to be confirmed in subsequent studies.
Published online: December 7, 2021.
Potential conflicts of interest: Please see https://www.psychiatrist.com/jcp/about-dr-andrade.
Funding/support: None.
3. Andrade C. Predictors of 6- and 12-month relapse after stopping electroconvulsive therapy: critical considerations, including overfitting in regression and confounding in follow-up studies. J Clin Psychiatry. 2021;82(4):21f141741.
Comments
Post a Comment