Artificial intelligence for predicting orthodontic patient cooperation: Voice records versus frontal photographs

INTRODUCTION

Patient cooperation refers to the harmony between the patient and the doctor throughout the course of the treatment and the patient’s compliance with the doctor’s orders. In terms of orthodontics, a compatible patient is one who maintains good oral hygiene, continues to keep all of his/her appointments, does not break brackets, uses elastic correctly and properly, and pays attention to his/her diet.^[1] If the patient fails to meet any of these requirements, the treatment will be prolonged, the targeted occlusion will not be achieved, or the treatment will be terminated before these goals are met. In other words, the orthodontist’s knowledge, skills, and experience, as well as patient compliance and cooperation, are required to achieve an ideal occlusion at the end of treatment.^[2-4]

In the present literature, orthodontists employ scales such as the orthodontic patient cooperation scale (OPCS) ^[4] or clinical cooperation evaluation^[3] to estimate patient participation.^[5] However, in recent years, the use of artificial intelligence (AI) in dentistry and medicine has become widespread.^[6,7] In this case, the following question comes to mind: Can an alternative method be created for the above-mentioned cooperation scales using AI in patient cooperation estimation?

The term “artificial intelligence (AI)” refers to computing systems that have human-like abilities such as learning and problem-solving. AI can complete some tasks more efficiently than humans because it is less prone to errors. Furthermore, AI works faster than human intelligence, with performance not expected of any human, and it can work accurately and effectively 24 h a day, seven days a week.

Making a choice or decision about any subject can be influenced by a person’s emotions or prejudices. By eliminating human prejudices and emotions from the equation, AI, on the other hand, makes evaluations based on numerical data. When assessing a patient’s cooperation at the start of treatment, the orthodontist’s subjective criteria may result in a more error-prone evaluation. In this situation, AI’s cooperation prediction can be more objective and consistent.

Photographs of a person’s face can be used to learn about their psychological characteristics. This is called the physiognomy of the face.^[8] Some psychologists believe that a person’s face can provide more quantitative information about their psychological characteristics than questionnaires and neuropsychological tests.^[9] Modern face analysis techniques allow for the assessment of not only psychological traits but also general health conditions. For instance, it is known that there are more than 700 genetic disorders that affect the structure and features of the face, and specialized software has been created to help diagnose these conditions.^[10,11]

Voice is a crucial communication tool that enables the speaker to express himself/herself. The human brain analyzes a person’s face, voice, and other physical characteristics using unidentified criteria when forming first impressions and making personality judgments.^[12,13] These data points could allow an orthodontist to make a preliminary assessment of the patient’s cooperation. This evaluation, however, cannot be confirmed due to its subjectivity and lack of reproducibility. According to the findings of numerous previous studies, researchers agree that personality traits can be successfully learned through voice.^[14,15]

The primary objective of the study is to evaluate the performance of various AI algorithms using frontal photographs and voice recordings. The secondary purpose is to compare the effectiveness of voice recording data versus photographic data in training AI algorithms.

MATERIAL AND METHODS

The research protocols of this study were approved by the Afyonkarahisar Health Science University Clinical Research Ethics Committee (ID = 2022/30). The study included 237 (147 females and 90 males; mean age 14.94 ± 2.4) systemically and mentally healthy patients undergoing fixed orthodontic treatment. The inclusion criteria were no prior orthodontic treatment history and no prior orthodontic treatment history, and undergoing at least 12 months of ongoing fixed orthodontic treatment were requirements for inclusion.

The doctors filled out an OPCS form at the 12^th month of treatment to determine the patients’ level of cooperation. This scale, created by Slakter et al., has ten criteria that describe patient behaviors that are frequently taken into account when evaluating the level of patient cooperation [Figure 1].^[4]

Close

Figure 1:

Orthodontic patient cooperation scale.

Export to PPT

OPCS forms were filled out for 286 patients who had been pre-evaluated and met the study’s eligibility criteria. Patients who received a total score of 60% or higher were considered cooperative (34 males and 98 females), while those who received a score of 40% or less were considered non-cooperative (56 males and 49 females). Patients with a score between 41% and 59% (25 females and 31 males) were not included in the study because they were on the borderline.

Routine extraoral photographs were taken with a standardized protocol in our orthodontic clinic. The images were taken with a Canon (EOS 60D, Japan-Tokyo) camera (shutter speed: 1/125, aperture: F22, ISO: 100) from a distance of 1.5 m and under identical lighting conditions. Resting frontal photographs of the patients, which were standardized as described above, were used in the present study.

Patients were asked to read aloud a standard text to obtain voice recordings that were taken in a quiet room using an IPhone (Apple 12 pro max, USA-California). Patients were not allowed to read the text aloud a second time to standardize the records, and only the recording from the first reading was analyzed.

The images were processed using nine different convolutional neural network (CNN) models. The architecture and number of layers of each of these models (VGG16, ResNet50 V2, ResNet101 V2, ResNet152 V2, InceptionResnetV2, Xception, MobileNet V2, NasNetMobile, and DenseNet) varies. CNN continuously analyzes the information in the data set using its mathematical and logical structures to generate a result that is similar to human conclusions.^[16] Each layer tries to learn some qualities by applying different filters to the image sample, and one layer transmits the image it processes to the next layer to learn other characteristics. The final layer divides the samples into two categories based on the retrieved features: Cooperative and non-cooperative.^[17]

In the present study, the models were first trained on the ImageNet dataset using the transfer learning technique to get the best performance out of the dataset, and the knowledge they acquired was then applied to the existing dataset’s training and classification.

For audio recording analysis, the machine learning algorithms K-nearest neighbors (KNN), logistic regression, random forest classifier, support vector machine (SVM), Decision Tree classifier, Gaussian Naive Bayes, linear discriminant analysis, AdaBoost classifier, Gradient Boosting classifier, XGBoost classifier, Extra tree classifier, voting classifier, Stacking classifier, and YAMNet (Yet Another Mobile Network), which are a CNN model, were used. YAMNet is a deep neural network that has been pre-trained with hundreds of sound spectrograms and is capable of predicting 521 different sound categories. Before processing audio input, YAMNet transforms it into spectrograms. It, then, analyzes the spectrogram images and identifies the sounds based on the information acquired from the analysis.

The models were created in the integrated development and learning environment using the Python programming language. A total of 80% of the sample was used for training, and 20% for testing. The models’ success was determined by their classification accuracy. Analysis for precision, recall, and F1 score was also used to assess the findings. The receiver operating characteristic (ROC) curve and confusion matrix of each model of the AI models used for photographs were examined comparatively.

RESULTS

The accuracy, precision, recall, and F1-score values of nine different CNN models trained with photos are shown in [Table 1]. The ROC curves which graphically illustrated each model’s efficacy, are also shown in [Figure 2]. Based on overall success, Xception (66.0%) and DenseNet121 (66.0%) were the two most effective CNN models. The success rate for ResNet101V2 was the lowest (48.0%). The success rate of the remaining five models’ were similar.

Close

Figure 2:

Receiver operating characteristic curves of the models for photographs (blue curves); angular dashed lines (black); (a) Xception, (b) DenseNet121, (c) ResNet50V2, (d) MobilNetV2, (e) ResNet152V2, (f) NasNetMobil, (g) InceptionResNetV2, (h) VGG16, and (i) ResNet101V2. ROC: Receiver operating characteristic; VGG16: Visual geometry group16

Export to PPT

The confusion matrix results of all photo-trained models are shown in [Figure 3]. The results of the confusion matrix showed that the ResNet50V2 (70.4%) model outperformed all others in predicting cooperative patients. The best models for predicting uncooperative patients (57.9%) were Xception and DenseNet121. The success rates of the remaining models for both cooperative and non-cooperative patients varied.

Close

Figure 3:

Confusion matrix results of models for photographs (a) Xception, (b) DenseNet121, (c) ResNet50V2, (d) MobilNetV2, (e) ResNet152V2, (f) NasNetMobil, (g) InceptionResNetV2, (h) VGG16, and (i) ResNet101V2. ROC: Receiver operating characteristic; VGG16: Visual geometry group16 (VGG16)

Export to PPT

The accuracy rates of the voice-trained models are shown in [Table 2]. Machine learning algorithms with the highest success rate were as follows: Linear discriminant analysis, KNN, SVM, Extra tree classifier, Stacking classifier), and YAMNet. The success rate of all of these five algorithms was 58.7%. The decision tree had the lowest prediction rate (41.3%).

In [Figure 4], the voice-model confusion matrix results were shown. Linear discriminant and AdaBoost classifier (56.2%) models, performed the best in terms of predicting cooperative patients, according to the results of the confusion matrix. The extra tree classifier and stacking classifier were the most successful models in predicting non-cooperative patients (70.0%). The prediction rates of other models’ varied for both cooperative and non-cooperative patients.

Close

Figure 4:

Confusion matrix results of the YAMNet models and machine learning algorithms (a) YAMNet, (b) Linear discriminant analysis, (c) K-nearest neighbors, (d) Support vector machine, (e) Extra tree classifier, (f) Stacking classifier, (g) Extra gradient boosting classifier, (h) Logistic regression, (i) AdaBoost classifier, (j) Random forest classifier, (k) Gaussian NB, (l) Gradient boosting classifier, (m) Voting classifier, and (n) Decision tree classifier. YAMNet: Yet Another Mobile Network, Gaussian NB: Gaussian naive bayes

Export to PPT

DISCUSSION

OPCS or Clinical compliance evaluation (CCE) scales have been used in previous studies to assess patient cooperation.^[3,4,18] However, these scales are composed of a series of clinically scored questions that assess some of the patients’ behaviors. The clinician’s qualitative impressions of the patient may influence the scale results.^[19] Therefore, the present study was designed with the idea of developing a method that provides orthodontists with more quantitative and reproducible results in learning about patient cooperation.

Many research have shown that facial anatomy and voice carry important personal information.^[20-23] Personality analysis methods may now be performed with greater precision and reproducibility because of the advancement of AI technology and its application in the medical industry.^[7,24] In the literature, CNN models are commonly utilized in the processing of visual data for personality analysis, and successful results have been achieved. Mukhopadhyay et al. employed the CNN model in their study to characterize photographers based on their emotional state using facial images, and they had 76.6% success with this model.^[25] In another identical study, a CNN model with a different architecture was utilized for the same goal, and the CNN’s success rate was 80.5%.^[26]

One of the most critical factors influencing the outcome of deep learning studies is the architecture of CNN models. The variation in model architecture has a direct impact on model performance, and the success rate can alter. CNNs with three different structures, for example, were utilized in a study that predicts the most prominent personality qualities (extraversion, openness, agreeableness, and meticulousness) using 240 pictures. As a result, varying success rates of 77.2%, 81.5%, and 85.3% were achieved.^[27] Nine alternative CNN models were utilized to strengthen the reliability of the present study’s results and to determine the most effective CNN model.

The content and quality of the data set, like the model architecture, might influence the success rate. Allen-Zhu et al., for instance, suggested that when two separate data sets were processed by the same CNN model, different results were achieved.^[28] Only rest frontal images and text-to-speech recordings that did not contain any visible emotion were preferred to establish standardization in the present study’s data collection.

High success rates are attained while using YAMNet in a variety of sectors, including technology,^[29] health,^[30] and industry.^[31] For example, the accuracy of YAMNet for the COVID-19 diagnosis^[32] was >97%, while it was 82% for the Parkinson’s disease diagnosis.^[30]

We had to generate our own dataset in the present study to estimate patient cooperation because there was no pre-prepared dataset. Nevertheless, it is not always possible to produce an extensive data set in a single center. Creating mega datasets, particularly in the health industry, takes time and money. However, the invention of the transfer learning technique has aided in the abolition of this difficulty. Using pre-trained and highly successful model knowledge eliminates the need for mega data sets in this approach.^[33,34]

Maray et al. used ImageNet, a mega dataset, to perform transfer learning before employing AlexNet in the diagnosis of Alzheimer’s disease. As a result, they reported that AlexNet powered by transfer learning outperformed models trained using other methods.^[34] In another study, the classification of a small dataset using the transfer learning technique performed significantly better than the classification of big datasets using a standard CNN model.^[35] In the present study, a transfer learning technique was used with the ImageNet data set to improve the efficiency of model training. As a result, the Xception and DenseNet121 models achieved a 66.0% success rate. When the confusion matrix was analyzed, ResNet50V2 performed best in predicting cooperative patients, with a success rate of 70.4%, whereas Xception and DenseNet121 performed best in predicting non-cooperative patients, with a success rate of 57.9%. YAMNet was employed for audio data and had a success rate of 58.7%. In addition to YAMNet, a CNN, 13 commonly used machine learning algorithms were tested in voice data processing. Some of these algorithms, however, produced results similar to YAMNet.^[36-41] The present study’s relatively low success rate may be attributed to the employment of just standardized models on a specific issue, such as cooperation prediction. However, even better results may have been achieved by combining the models that were most successful in predicting cooperative and non-cooperative patients.

There are also studies in the literature that use voice recordings to evaluate personality or psychological traits. For instance, in the study by Liu et al., the scale selection pyramid network dataset was used to assess the performances of SVM, KNN, and logistic regression algorithms in defining personality traits.^[37] The success rate obtained using these algorithms was reported as 72.0% (SVM), 70.0% ( KNN), and 71.0% (logistic regression). Various machine learning algorithms and an CNN model were used in the present study to determine whether patients were cooperative or non-cooperative using voice data, and acceptable results were obtained when compared to the results of the previous studies.

The present study aimed to investigate the performance of AI, particularly deep learning, on patient cooperation prediction and to pave the way for further researches on this topic. The present study’s findings are intriguing, but more research using more complex models and data sets acquired from various centers is required.

CONCLUSION

The present study’s findings demonstrate that cooperation prediction can be made using AI in orthodontics. Some CNN models trained using patient photographs predicted cooperation with acceptable accuracy. Voice recordings, on the other hand, were not as effective as photographs in predicting cooperation. However, it is clear that both types of data can help the orthodontist predict cooperation to varying degrees.