Are artificial intelligence based chatbots reliable sources for patients regarding orthodontics?

INTRODUCTION

Traditional search engines (TSE), conversational intelligence virtual assistants (CIVA), and artificial ıntelligence-(AI-) based chatbots are all technological tools designed to provide information and assistance.^[1,2] Each of these tools serves different purposes and meets various user needs in the area of information access and interaction. However, the utilization of AI-based chatbots for delivering health-related information to patients can offer specific advantages over TSE and CIVA, particularly in certain scenarios.^[3,4]

AI-based chatbots possess several advantages, such as 24/7 availability, enabling patients to seek information at any time, providing instant responses to patient queries, and reducing the necessity to wait for appointments or responses from healthcare professionals.^[5] In the context of orthodontic practice, chatbots can educate patients about orthodontic procedures, assisting them in making informed decisions regarding their care. In addition, patients can use chatbots for quick clarifications or answers to minor concerns without the need for a formal appointment, thus saving time and potentially reducing healthcare costs. These devices also offer a private setting, avoiding potential embarrassment in face-to-face discussions. ^[2,3,6-8]

Although AI chatbots hold promise, their effectiveness must be supported by rigorous research and validation. Such research can provide a comprehensive perspective on the capabilities and limitations of AI-based chatbot technology when it comes to delivering sensitive orthodontic information, ultimately contributing to a more comprehensive understanding of their performance in healthcare contexts.

While there is a growing body of scientific studies investigating how accurately chatbots provide information to patients in various healthcare fields, to the best of our knowledge, there is currently no research examining the accuracy and effectiveness of AI-powered chatbots in responding to frequently asked questions by patients in the field of orthodontics.^[9-14] Therefore, the aim of this study was to evaluate the accuracy and efficiency of three popular AI-powered chatbots, ChatGPT-4 (1), Microsoft Copilot (2), and Google Gemini (3), in responding to common inquiries posed by orthodontic patients.

MATERIAL AND METHODS

This study did not require ethical approval due to its publicly available nature. We identified a total of 20 questions that patients commonly asked about orthodontic treatment and posed them to three AI-based chatbots: ChatGPT-4, Microsoft Copilot, and Google Gemini [Table 1]. When formulating the queries, we deliberately avoided simple models, opting instead for more scenario-based questions to evaluate the capabilities of AI systems. We accomplished this by identifying the most frequently asked orthodontic queries on the “Quora Digest” platform, facilitating a comprehensive evaluation of chatbot performance in addressing real-world orthodontic problems. This platform is widely used for sharing and discussing trending topics, making it an ideal source for gathering questions that are representative of the types of inquiries users typically seek answers to in everyday situations.

The responses provided by the three platforms were recorded and assessed for accuracy, comprehensiveness, and clarity. Ten orthodontic experts, all of whom have extensive clinical experience and are regularly exposed to similar cases in their professional practice, reviewed the information to assess its clinical validity. To evaluate the accuracy and completeness level of each response, we employed a modified Likert scale (LS), customized for this study. This modified LS included five response options: (1) the chatbot provided accurate information, and the response covered all relevant aspects of the question; (2) the chatbot provided accurate information, but the response did not cover all relevant aspects of the question; (3) the chatbot provided inaccurate information, but the response covered all relevant aspects of the question; (4) the chatbot provided inaccurate information, and the response did not cover all relevant aspects of the question; and (5) the chatbot couldn’t provide an answer.

To assess the clarity and flow of the responses, we employed the Global Quality Scale (GQS) by Bernard et al.^[15] [Table 2]. The score assigned to each device was calculated by averaging the ratings provided by the experts for each response.

RESULTS

Statistical Package for the Social Sciences (SPSS) for Windows, version 23.0 (SPSS Inc., Chicago, IL, USA), was utilized for all statistical analyses, with the level of significance set at P < 0.05. The normality of the data was assessed through the Q-Q plot diagram and Shapiro–Wilk test, revealing that all variables were normally distributed. Consequently, we conducted a one-way analysis of variance and post-hoc analyses.

The mean LS and GQS scores for each query of the three chatbots are presented in [Table 1]. The results indicated that ChatGPT-4 (1.69 ± 0.10) and Microsoft Copilot (1.68 ± 0.10) achieved significantly higher Likert scores than Google Gemini (2.27 ± 0.53) (P < 0.05). However, the GQS scores were as follows: 4.01 ± 0.31 for ChatGPT-4, 3.92 ± 0.60 for Google Gemini, and 4.09 ± 0.15 for Microsoft Copilot. Notably, no significant differences were observed among the three chatbots in terms of GQS (P > 0.05) [Tables 3 and 4].

Furthermore, we found moderate negative correlations between LS and GQS scores for ChatGPT-4 (r = −0.250), mild negative correlations for Microsoft Copilot (r = −0.088), and moderate positive correlations for Google Gemini (r = 0.466).

DISCUSSION

Over the years, AI has been a driving force in advancing digital healthcare.^[16,17] AI-powered tools in the field of dentistry have proven to be valuable for analyzing medical images, aiding in the diagnosis of conditions such as dental caries, periodontitis, and implants, as well as assisting in the planning of oral and maxillofacial surgeries.^[18-22]

In the field of orthodontics, neural networks can help diagnose and plan treatments, mark cephalometric landmarks, analyze anatomy, assess growth and development, and evaluate treatment outcomes.

In the present study, we examined the performance of three individual AI-powered chatbot systems – ChatGPT-4, Microsoft Copilot, and Google Gemini – in providing responses to orthodontic-related queries from 10 orthodontic experts. Our evaluation criteria included LS to assess the accuracy and completeness of responses and GQS to evaluate the clarity of the responses.

Microsoft Copilot, Google, Google Gemini, and ChatGPT-4 were selected due to their current prominence and widespread use in the field of conversational AI. These models represent the most advanced and commonly accessed systems at this time, making them highly relevant for the purpose of our study. We have also highlighted the importance of evaluating the leading AI systems in real-world contexts, as these are the models that users are most likely to engage with when seeking information on specialized topics such as orthodontics. This adjustment aims to reinforce the reasoning behind our selection and to make the decision clearer to the reader.

ChatGPT-4 is based on a large pre-trained language model that can create fluent and varied texts using a deep neural network. Microsoft Copilot is a web search engine that uses different natural language processing (NLP) techniques, such as understanding queries, retrieving documents, and summarizing them to give clear and informative answers. However, Google Gemini is still in the early stages of development and is mainly designed to create creative content such as poetry and lyrics.^[10,12]

The results of our study indicate that there were no significant differences between the responses provided by the three chatbots, ChatGPT-4, Google Gemini, and Microsoft Copilot, for the three straightforward, patterned questions posed (N:1, 3, 6) (P > 0.05) [Table 1]. These questions revolved around common orthodontic issues, including discomfort caused by braces, broken brackets, and loose brackets. This consistency suggests that these chatbots, despite differences in their underlying algorithms or training data, are able to provide uniform and reliable information for straightforward orthodontic queries. This finding is promising as it implies that users can expect consistent and accurate responses to their simple patterned questions regardless of the specific chatbot they interact with.

When the LS scores of 3 chatbots are examined, ChatGPT-4 (1.69 ± 0.10) and Microsoft Copilot (1.68 ± 0.10) received mean scores falling within the range of categories 1–2. This indicates that these chatbots generally provided accurate information along with comprehensive responses. However, each platform exhibited strengths in different areas: ChatGPT-4 excelled in generating comprehensive responses, while Microsoft Copilot offered concise and clear explanations. In contrast, Google Gemini (2.27 ± 0.53) received mean scores falling in categories 1–3, signifying a wider variability in its performance compared to the other two chatbots. This variability indicates that users should be cautious when using this platform, as Google Gemini occasionally offers inaccurate information and less comprehensive answers.

ChatGPT-4 has demonstrated a good ability to provide coherent and comprehensive answers to orthodontic questions, regardless of their complexity. Microsoft Copilot, too demonstrated clarity in its responses, providing comprehensive informative information. The superior performance of ChatGPT-4 and Microsoft Copilot could be attributed to their advanced NLP capabilities, which enable them to understand complex queries and generate relevant and coherent responses. However, Google Gemini, while user-friendly in its interface, is a chatbot that uses a simpler approach of matching keywords and phrases from the query to a predefined database of answers. This method may limit the ability of Google Gemini to handle queries that require more reasoning and synthesis. In addition, the responses provided by Google Gemini sometimes seemed to veer slightly off-topic or include extraneous information, which could potentially confuse users. These results are consistent with previous research, which has shown that large language models such as ChatGPT-4 and Microsoft Copilot are capable of generating accurate and informative responses to a wide range of queries. ^[22-25]

Regarding response flow and clarity, as evaluated through GQS scores, the orthodontic experts perceived all three chatbots in a similar manner. This observation implies that, despite variations in response accuracy and comprehensiveness, the clarity and flow of explanations and information presentation remain consistently uniform among the platforms examined. It is important to note that these models are still under development, and their performance can vary depending on the specific task and the quality of the training data. As AI continues to advance, these platforms have the potential to play a pivotal role in enhancing patient education and information dissemination in the field of orthodontics and healthcare more broadly. The ongoing refinement of AI algorithms will likely lead to even more accurate, comprehensive, and user-friendly responses, further bridging the gap between patients and reliable medical information.

It is important to remember that the field of AI is dynamic, and the accuracy of chatbot responses may evolve over time as models are updated. This study is limited by the specific sample of queries used, the potential for evolving AI capabilities, and the subjectivity of expert review.

CONCLUSION

In summary, while AI-based chatbots can provide valuable information and guidance, their accuracy depends on the quality of their training data, the involvement of medical experts, continuous updates, and clear communication of limitations. Therefore, users should always verify medical information with qualified professionals before making decisions about their orthodontic treatment.