Assessing the impact of the COVID-19 pandemic on the quality of evidence reported in the leading orthodontic journals

INTRODUCTION

In the field of medical and dental research, randomized controlled trials (RCTs) are considered the “gold standard” for clinical guidelines and one of the most important sources of evidence.^[1-3] Significance of RCTs findings depends on their validity, which is determined by their design, methodology, and execution.^[4] RCTs ideally entail a study design that eliminates bias and guarantees more valid and valuable data compared to other study designs. They, therefore, present a reliable method for assessment of the effectiveness of therapeutic modalities and medications. Over the years, several objective scales, individual markers, and checklists for assessing the quality of RCTs have been developed.^[5] A systematic review reported that there were around 21 scales for assessing the quality of RCTs, with varying validity and reliability standards.^[6-8]

The quality of RCT’s can also be assessed using the Consolidated Standards of Reporting Trials (CONSORTs) tool. The CONSORTs statement was first published in 1996; a revised statement was published in 2001 and later in 2010.^[9-11] CONSORT is a protocol developed to guide researchers not only on how to identify problems arising from conducting RCTs but also to report, fully and clearly, the results yielded by the research, thereby facilitating RCT reading and quality assessment.^[9-11] This statement consists of a 32-item checklist and a flow diagram in which investigators are encouraged to report on the various aspects of how RCTs were conducted. Some important items include sample size calculation, randomization, blinding, statistical elements, subgroup analyses, and confounding/stratification. CONSORT also consists of a flow diagram that provides a summary of the process of how the RCT was conducted, including the enrolment, allocation, follow-up, and analysis of the participants.^[9] Although the reporting of RCTs has recently improved, particularly in journals that have adopted the CONSORT statement (post-CONSORT), the reporting of certain items remains suboptimal even when the CONSORT guidelines are seemingly followed.^[12-14] Recommendations are offered to authors and researchers by many editors to follow structured-format and to comply with the CONSORT guidelines to enhance RCTs reporting.^[12,13,15]

The onset of COVID-19 pandemic had a devastating impact all around the world. It had affected many aspects of life in a way that was unprecedented in modern history and the consequences are still not fully recognized. Higher education institutions and universities were forced to adapt to the rapidly changing situation. Research institutions were facing considerable challenges in managing research operations. The mandatory social distancing requirements were difficult to apply in the research setting, particularly in areas requiring bench work and human subjects, as well as fieldwork. Most of this has significantly affected scientists, faculty, research scholars, graduate students, and scholarly activities in general. The career plans of many scientists and researchers were at risk due to the sudden interruption in their research plans by the pandemic.^[16]

In the dental literature, very few numbers of studies have reported the quality of RCTs including clinical trials in orthodontics^{[12-14,17-19]} and no study has been carried out to evaluate the effect of the COVID-19 pandemic on the quality and quantity of RCTs in orthodontics research. The aim of this study was to assess the quality of research reporting of RCTs in the orthodontic literature published in three of the highest impact factor orthodontic journals (the American Journal of Orthodontics and Dentofacial Orthopedics [AJODO], the Angle Orthodontist [AO], and European Journals of Orthodontics [EJO]) and also to assess the effect of COVID-19 pandemic on the quality of reporting of the latest orthodontic literature published in these orthodontic journals.

MATERIAL AND METHODS

A retrospective analysis was conducted by reviewing and hand-searching all articles published in AJODO, EJO, and the Angle Orthodontist (AO) from the year 2018 to 2022. Articles that reported RCTs were identified. Identification of the trials was done by searching for the keywords “randomized,” “clinical trial,” and then retrieving the full text for all the articles. A buffer period was set from September 2019 to August 2020 to allow for the proper classification of the articles into two groups (pre-COVID-19 [January 2018–August 2019] and post-COVID-19 [September 2020– March 2022]).

Critical appraisal and investigation of the quality of all included RCTs were performed. All RCTs were read in full and assessed using the CONSORT checklist [Supplementary Figure 1].^[20] A score of “yes, no, or not applicable (NA)” was assigned to all 37 items for each trial according to the compliance and adequacy of information description as judged by two independent investigators. The total score for each trial was calculated and converted to a percentage using the equation:

Total score = (total number of “Yes”/[37-total number of “NA” items]) × 100.^[11] Discrepancies between both reviewers were resolved by discussion and agreement. Additional data items were extracted from each article including the number of authors, year of publication, country, setting, number of groups, type of orthodontic treatment, and statistical testing used.

RESULTS

Correlation test showed high reliability of 0.99 and 0.96 suggesting overall excellent inter-rater agreement and reliability. A total of 117 RCTs were included in the study. The sample was classified as follows: 63 pre-COVID studies (53.8%) and 54 (46.2%) post-COVID studies, published in three orthodontics journals (27 [23.1%] in AJODO, 45 [38.5%] in AO, and 45 [38.5%] in the EJO). Mean percentage of CONSORT compliance was 84.51% ± 14.34%. Eleven (9.4%) of the studies were from USA, 53 (45.3%) from Europe, while 53 (45.3%) were from other countries. Forty-six (39.3%) of the trials were reported by four authors or less, 66 (56.4%) by 5–8 authors, and only five (4.3%) were reported by more than eight authors. The highest percentage of the trials 36, 30.8% were published in 2018, 27, 23.1% were published in 2019, 22, 18.8% were published in 2020, and 20, 17.1% were published in 2021 [Table 1 and Figure 1].

Close

Figure 1:

(a) Journals where the studies were published, (b) countries of the included studies, (c) number of authors in the included studies, and (d) number of studies published per year. AJODO: American journals of orthodontics and dentofacial orthopedics, AO: Angle orthodontist, EJO: European journal of orthodontics.

Export to PPT

Independent t-test was performed to compare RCTs pre/post-COVID across CONSORT percentages. It showed no statistically significant difference as the mean compliance was 84.96 ± 16.01 for the pre-COVID studies and 83.99 ± 12.25 for post-COVID (P = 0.711) [Table 2].

CONSORT compliance scores were compared among the three journals for pre/post-COVID using an independent t-test. Statistically significant difference was recorded for the AO where CONSORT percentages were higher for the post-COVID studies (81.45 ± 15.53) than pre-COVID (70.27 ± 16.03) (P = 0.022). For the EJO, a statistically significant difference (P < 0.001) was also recorded where CONSORT percentages were higher for the pre-COVID studies (93.23 ± 9.36) than post-COVID (81.93 ± 7.85). For the AJODO, there was no significant difference [Table 3 and Figure 2].

Close

Figure 2:

Mean Consolidated Standards of Reporting trials scores by journal (pre/post-COVID). CONSORT: Consolidated standards of reporting trials.

Export to PPT

To assess the association of different factors with the CONSORT score, multiple linear regressions were used. There was a statistically significant association of CONSORT scores with journals where the trials were published. As compared to AJODO, AO had a lower CONSORT percentage by an average of 18.81, P < 0.001. All other factors showed no statistically significant association with the CONSORT percentage [Table 4].

DISCUSSION

Orthodontic literature is a critical source for evidence-based practice and decision-making for all practitioners. The ability to judge the quality of research reporting is necessary to allow clinicians and researchers to reach a valid conclusion and make a correct decision. A comprehensive evaluation of the profile of the most recent orthodontic research and literature will help the orthodontic community update their information on its reliability and effectiveness in answering clinical- and practice-related questions.^[21]

The COVID-19 outbreak in early 2020 tremendously influenced all life situations including academia.^[22] The higher education system, including research output, was greatly affected on different scales.^[23] The aim of this study was to assess the effect of COVID-19 pandemic on quality of research reporting of the most recent orthodontic literature published in three of the highest impact factor orthodontic journals.

Lempesi et al., in their study, concluded that the methodological quality of RCTs in prominent orthodontic journals was below expectations, especially when compared to other dental and medical periodicals.^[24] It is worth mentioning that while the CONSORT criteria have been used for RCTs analysis by more than 800 international periodicals of different specialties, in orthodontics, only some journals such as the EJO and AJODO decided to endorse the use of these criteria to accept RCTs for publication. Sandhu et al., reported significant improvements in the quality of RCTs after these journals began to adopt CONSORT criteria.^[25] Such progress was particularly noticed in articles published from 2010 on, when CONSORT was revised.

In the current study, a total of 117 RCTs (63 pre-COVID and 54 post-COVID) were included in the assessment. The mean percentage of CONSORT compliance was 84.51% ± 14.34% for articles published from 2018–2022. Previously, Bearn and Alharbi^[12] investigated whether authors in the orthodontic field of research report RCTs adequately as defined by the CONSORT statement by reviewing the orthodontics RCTs published between 2008 and 2012. They reported a mean CONSORT score of 51.7% and an overall compliance increase from 47.8 to 56.3% between 2008 and 2012.^[12] Similarly, Kloukos et al. assessed the quality reporting of RCTs published in prosthodontics and implantology journals, and they reported a mean modified CONSORT score of 60.9% to –80.6% across the journals.^[26]

While screening and during the process of data extraction for the current study, certain trends of research focus in orthodontics RCTs were noticed. For example, during 2018, most of the studies centered around evaluating the effect of vibration devices such as AcceleDent Aura Appliance on the acceleration of orthodontic tooth movement, space closure, treatment duration, and occlusal outcomes as well as the low-level laser therapy and its effect on pain reduction and repair of orthodontically induced inflammatory root resorption.

The findings of the current study indicated that there was no statistically significant difference between pre- and post-COVID CONSORT scores with a mean compliance of 84.96 ± 16.01 for the pre-COVID studies and 83.99 ± 12.25 for post-COVID (P = 0.711) implying that the pandemic did not affect the overall quality of reporting of RCTs across the three orthodontic journals. Therefore, although COVID-19 had a significant impact on the orthodontic practice and education in general, it did not seem to similarly affect the research output and specifically the RCTs reported in the top orthodontics specialty journals.

Results also showed that AJODO recorded the highest CONSORT mean score, followed by the EJO, while the least compliance score was recorded for AO by an average of 18.81, P < 0.001 when compared to AJODO. Similar findings were reported by Alharbi and Almuzian, who stated that AJODO scored the highest among four other orthodontics journals with regard to the quality of reporting RCT abstracts for the period of 2012–2017.^[27] On the other hand, Bearn and Alharbi^[12] reported the highest CONSORT compliance score for the Journal of Orthodontics (73%) as compared to the AJODO (53.9%), EJO (48.9%), and the lowest was achieved by AO (44%) which was attributed to the fact of JO strict endorsement of CONSORT statement while AO did not.

CONCLUSION

The quality of reporting of RCTs in orthodontic journals was not affected by the pandemic. AJODO had the highest CONSORT compliance score while the AO had the least compliance score.