Evaluation of quality and reliability of videos about orthodontics on TikTok (DouYin)
How to cite this article: Meric P, Kılınç DD. Evaluation of quality and reliability of videos about orthodontics on TikTok (DouYin). APOS Trends Orthod, doi: 10.25259/APOS_141_2021
TikTok is the world’s most popular short video application, where orthodontics-related content is viewed billions of times. This study aimed to evaluate the quality and reliability of video content about orthodontics on TikTok.
Material and Methods:
TikTok mobile application was searched for videos with the hashtag “#orthodontics” on July 9, 2021. The content and quality of the first 150 videos were evaluated using the global quality score (GQS) and reliability score (modified DISCERN tool) which is an evaluation tool for health information on social media.
The mean of the two researchers’ GQS values and reliability score was 2.1 ± 1.1 and 1.2 ± 1.4, respectively. There was no statistically significant difference between the median values of GQS and reliability score according to categorical descriptive variables (P > 0.050). A statistically significant weak positive correlation was found between like no. and reliability scores (r = 0.245; P = 0.037). Most of the videos were uploaded by orthodontists (72.6%).
Both the quality and reliability of the content about orthodontics on TikTok were very low. Even the most of the accounts were professionals, they did not prefer to give scientific information due to the medium’s own dynamics and target population’s likes.
The comprehension of social media’s potential role in orthodontics can make us understand the motivations, expectations, and experiences of patients. Social media reflects cultural and social trends and those could affect treatment demand and satisfaction. A large number of young patients undergo orthodontic treatment to adapt to social norms and cultural norms imposed by the modern perception of beauty. For this reason, orthodontists should understand how their patients perceive orthodontic treatment and their experience with it. In addition, the potential impact of social media on orthodontics is of great importance not only for patients but also for clinicians, researchers, and research sponsors.
Freedom in the social media environment provides individuals with unlimited sharing opportunities. In this way, an endless accumulation of knowledge is created in social media, but the lack of any control mechanism of the information in the system brings “information pollution” with it. The effectiveness of the internet and social media in improving patient knowledge during treatment is in interest in recent years. It has been presented that orthodontic patients use the internet frequently for the topics they are curious about during their treatments. However, this approach may affect the treatment process positively or negatively depending on the quality and reliability of the information accessed. Studies have shown that orthodontic patients are more likely to keep in mind the information presented in visual and auditory format. Transmission in visual form is more effective in communicating information to adolescents. At this point, YouTube (YouTube, San Bruno, CA) has been preferred due to its visual and auditory information compared to other social media platforms. Nevertheless, TikTok (named as “DouYin” in China, ByteDance, Beijing, China) is a booming short video application (app) offering new potential for spreading healthcare-related information. Short videos have the potential to stimulate interest in content, and the entertaining approach in TikTok videos makes learning more fun.
TikTok app is used to make different types of short-form videos ranging from 15 s to 3 min in genres such as dance, comedy, and education. TikTok reported nearly 800 million global active users. About 42% of TikTok users are between the ages of 18 and 24 and approximately 27% of frequent users are aged 13–17 years. The mentioned age groups constitute the majority of the target group for orthodontic treatment. TikTok has a larger and more diverse audience than traditional “follower”-based social media models. Because videos are listed by hashtags, search results are ordered directly by the number of likes received per post.
At the time of preparing this study, visual content under various hashtags such as orthodontics, orthodontist, and brackets had received 658.2 million, 1.2 billion, and 1.3 billion views, respectively.
It has been presented that TikTok provides a stronger user engagement sense than other social media applications and platforms in terms of social presence and immersion. In addition to the positive aspects of using short video applications for health services, some negative effects have also been reported in previous studies. For example, some researchers have stated that TikTok is positioning itself to focus on interesting videos rather than serious professional content. As a result of the widespread use of short video apps as sources of health information, the bright and dark sides of the user experience on those apps are debatable.
Information about health care on YouTube is subjected in various previous studies and has been found as “not reliable and not qualified” due to the lack of peer review and detailed evaluation which could result in the spreading of erroneous information.[13,14] Although studies are evaluating the reliability and quality of video content about orthodontics on YouTube, there is not a study evaluating the content about orthodontics on TikTok. This study aimed to evaluate the quality and reliability of video content about orthodontics on TikTok.
MATERIAL AND METHODS
Ethical approval was not required as no human material was used in the study. TikTok mobile application was searched for videos with the hashtag “#orthodontics” on July 9, 2021. The top 150 most-liked videos were simultaneously favorited by two 10 years experienced orthodontists and then evaluated by the researchers blinded to each other. The content and the quality of the first 150 videos for the “#orthodontics” hashtag were assessed using the global quality score (GQS) [Table 1] and reliability score (modified DISCERN tool) [Table 2] which are assessment tools for health information on social media. The search was filtered as “content-type: Videos; sort by: Most liked; and date posted: All the time.”
|Score||Global quality score description|
|1||Poor quality, poor flow of the site, most information missing, not at all useful for patients|
|2||Generally poor quality and poor flow, some information listed but many important topics missing, of very limited use to patients|
|3||Moderate quality, suboptimal flow, some important information is adequately discussed but others poorly discussed, somewhat useful for patients|
|4||Good quality and generally good flow, most of the relevant information is listed, but some topics not covered, useful for patients|
|5||Excellent quality and excellent flow, very useful for patients|
|Score||Reliability scorea (Adapted from DISCERN)|
|1||Are the aims clear and achieved?|
|2||Are reliable sources of information used?|
|3||Is the information presented balanced and unbiased?|
|4||Are additional sources of information listed for patient reference?|
|5||Are areas of uncertainty mentioned?|
Exclusion criteria were as follows: Videos in languages other than English videos unrelated to orthodontics, duplicated videos, and videos with no audio. After the exclusion of 77, not suitable videos, the remaining 73 videos were assessed [Figure 1].
Upload year, uploader type (orthodontist-non-orthodontist), uploader gender, like the number, share number, and comment number parameters were recorded as descriptive properties for each video.
Data were analyzed with IBM SPSS V23. Conformity to normal distribution was evaluated by Shapiro–Wilk and Kolmogorov–Smirnov tests. The Mann–Whitney U-test was used to compare the scores that were not normally distributed. For the comparison of more than 2 independent samples, the Kruskal–Wallis test was used to compare the scores that were not normally distributed. The agreement between the raters was examined by the intraclass correlation coefficient (ICC). Spearman’s rho correlation coefficient was used to examine the relationship between not normally distributed quantitative data. Analysis results were presented as mean ± standard deviation and median (minimum-maximum) for quantitative data, and frequency and percentage for categorical data. The significance level was taken as P < 0.05.
There was a statistically significant-excellent agreement between the two researchers in terms of GQS (ICC = 0.921; P < 0.001). Likewise, there was a statistically significant perfect agreement between the two researchers in terms of the reliability score (ICC = 0.931; P < 0.001) [Table 3].
|ICC (%95 CI)||P-value|
|Reliability score||0.931 (0.892–0.956)||<0.001|
Of the videos not included in the study, 64.9% were unrelated content and 31.2% were out of the English language. It was found that 63% of those included in the study were uploaded in 2021, 16.4% were about elastic ligature color, 57.5% were uploaded by men, and 72.6% of the uploaders were orthodontists [Table 4].
|Video language other than English||24||31.2|
|Debonding of brackets||4||5.5|
|Comment on a previous treatment||1||1.4|
|Uploader type 1|
The average number of comments was 4113.5, the average number of likes was 424,347.9, and the average number of shares was 3665. The mean GQS and reliability score of the two raters were 2.1 and 1.2, respectively [Table 5].
|GQS researcher 1 (PM)||2.1||1.1||2.0||1.0||5.0|
|GQS researcher 2 (DDK)||2.0||1.1||2.0||1.0||5.0|
|Reliability score researcher 1 (PM)||1.2||1.4||1.0||0.0||5.0|
|Reliability score researcher 2 (DDK)||1.3||1.5||1.0||0.0||5.0|
|Mean reliability score||1.2||1.4||1.0||0.0||5.0|
There was no statistically significant difference between the median values of GQS and reliability score according to categorical descriptive variables (P > 0.050) [Table 6].
|Mean±SD||Med. (Min.-Max.)||Mean±SD||Med. (Min.-Max.)|
|2019 and 2020||2.17±0.94||2.00 (1.00–4.00)||1.35±1.31||1.00 (0.00–4.50)|
|2021||2.04±1.15||1.75 (1.00–5.00)||1.15±1.49||1.00 (0.00–5.00)|
|Ligature color||1.46±0.45||1.50 (1.00–2.00)||0.71±0.72||0.75 (0.00–2.50)|
|Bracket failure||2.10±0.89||2.00 (1.00–3.00)||0.80±0.84||1.00 (0.00–2.00)|
|Diet recommendation||1.88±0.52||2.00 (1.00–2.50)||0.94±0.62||1.00 (0.00–2.00)|
|Dental impression||1.86±1.07||1.00 (1.00–3.00)||1.00±1.29||0.00 (0.00–3.00)|
|Orthodontic appliance||1.89±0.89||2.00 (1.00–3.50)||0.89±0.96||1.00 (0.00–2.50)|
|Orthodontic treatment||2.90±1.52||3.00 (1.00–5.00)||2.30±1.86||2.00 (0.00–5.00)|
|Treatment progress||2.40±1.34||2.50 (1.00–4.50)||1.60±1.85||1.50 (0.00–4.50)|
|Other||2.41±1.28||2.25 (1.00–5.00)||1.59±1.87||1.00 (0.00–5.00)|
|Female||2.35±1.11||2.00 (1.00–5.00)||1.53±1.54||1.00 (0.00–5.00)|
|Male||1.89±1.02||1.50 (1.00–5.00)||1.00±1.29||1.00 (0.00–5.00)|
|Uploader type 1|
|Non–orthodontist||1.85±0.97||1.50 (1.00–4.50)||0.98±1.25||0.50 (0.00–4.50)|
|Orthodontist||2.18±1.11||2.00 (1.00–5.00)||1.32±1.48||1.00 (0.00–5.00)|
|Uploader type 2|
|Layperson||1.71±1.12||1.00 (1.00–4.50)||0.75±1.37||0.00 (0.00–4.50)|
|Orthodontist||2.18±1.11||2.00 (1.00–5.00)||1.32±1.48||1.00 (0.00–5.00)|
|Dental care professionals||2.06±0.73||2.00 (1.00–3.00)||1.31±1.03||1.25 (0.00–3.00)|
A statistically significant weak positive correlation was found between the comment no. and the reliability score (r = 0.235; P = 0.046). A statistically significant weak positive correlation was found between like no. and reliability score (r = 0.245; P = 0.037). A statistically significant weak positive correlation was found between share no. and reliability score (r = 0.304; P = 0.009). A statistically significant weak positive correlation was found between share no. and GQS (r = 0.242; P = 0.039) [Table 7].
There was a statistically significant positive correlation between GQS and reliability score values (r = 0.887; P < 0.001) [Table 8].
TikTok is a short video app that is popular all over the world. It has been downloaded more than 1 billion times all over the world. In their letter to the editor of the British Journal of Oral and Maxillofacial Surgery, the researchers underlined that TikTok is an application that cannot be overlooked by oral and maxillofacial surgeons. However, misleading or incorrect health advice in the content on TikTok can lead to undesirable dangerous results among the public. This paper was prepared with this sensitivity and motivation, and this study aimed to investigate the quality and reliability of the video content about orthodontics on TikTok. To the best of our knowledge, there is no other study evaluating the quality and the reliability of the video content about orthodontics on TikTok. That is why we did not have a chance to compare our findings one to one.
TikTok is the fastest growing social media application worldwide where short videos ranging from 15 s to 60 s are shared and amplified. With its availability in more than 150 countries the app is reported to have more than 800 million active users. TikTok algorithm lists the videos according to the high like number of audience engagement. This method is different from other video-based platforms like YouTube. YouTube has no limitation in the video length and in which users have more control to determine the following video they want to view. The majority of the TikTok users are young people where 63.5% of the users are reported under the age of 29. Considering that the age range where the application is common is under the age of approximately 30, it was thought that the fact that both researchers in our study were in their early 40s would not create a huge difference in perspective between the researchers and the users and audiences of the application. In addition, there was a statistically significant-excellent agreement between two researchers in terms of GQS value (ICC = 0.921; P < 0.001) and in terms of the reliability score (ICC = 0.931; P < 0.001).
In two recent studies that evaluated the content of TikTok videos, researchers evaluated the top 100 videos as related to search hashtags.[8,18] In our study, we evaluated the first 150 videos under the hashtag #orthodontics. The content under that hashtag had been viewed 658.2 million times on TikTok.
In this study, GQS and the modified DISCERN tool were used to evaluate the video content. In their study evaluating the content about orthodontics on YouTube, Kılınç and Sayar used the same tools for evaluation was in his study, Üstdal and Güney used another form of DISCERN but the same form of GQS. In addition to this, Zheng et al. also used DISCERN tool for the evaluation of TikTok videos about acne in their study.
The mean of the two researchers’ GQS values and reliability scores was 2.1 and 1.2, respectively, in our study. Üstdal and Güney evaluated YouTube videos with the same tools and found this as 2 for reliability score and 2.91 for GQS for poor content and 2.03 and 3.42 for rich content. Kılınç and Sayar’s findings were 2.8 for GQS and 2 for DISCERN. In addition, in their study on acne and TikTok, Zheng et al. also found a mean quality rating of 2.1. This finding was also consistent with our findings.
In this study, when uploaders were divided into two groups as orthodontists and non-orthodontist and the reliability and quality of the content they uploaded was evaluated; there was no statistical difference between orthodontists and nonorthodontists as being uploaders.
The remarkable difference between GQS and reliability ratings can be assumed that the content evaluated by the researchers was visually appealing but interpreted as unscientific. Again, despite the low DISCERN values, the relatively high GQS values in which video quality is evaluated may be because researchers evaluating visual content are more interested in written content in most of their professional lives. These findings reveal the necessity of the scientific world to develop a new scientific-based visual content evaluation scale as soon as possible. Furthermore, the very low results in the DISCERN rating can be interpreted as the lack of specified sources of information for both orthodontist and non-orthodontist installers, the lack of discussion of treatment risks, and the lack of reliable scientific sources for content.
Seventy-seven of the evaluated videos in this study were excluded regarding exclusion criteria. The remaining 73 videos’ average number of comments, number of likes, and number of shares were 4113.5 and 424,347.9 and 3665.0, respectively. Unrelated videos were generally dramatic videos in genres of humor, dance, and music.
In our study, categories of evaluated videos were as follows: Ligature color, orthodontic appliance, diet recommendation, dental impression, bracket failure, orthodontic treatment, treatment progress, bracket bonding, debonding of brackets, flossing, patient experience, public health, retainer, appliance care, brushing, comment on previous treatment, complication, enameloplasty, and orthodontic emergency.
Although some orthodontist accounts have written that they were board-certified orthodontists in the information section about themselves, of all the videos we evaluated in our study, only one user who also was an orthodontist with the nickname @thyrants differed “scientifically.” The user posted a video about four premolar extraction treatments and cited the following article “Kook YA, Park JH, Bayome M, Sa’aed NL. Correction of severe bimaxillary protrusion with first premolar extractions and total arch distalization with palatal anchorage plates. Am J Orthod Dentofacial Orthop. 2015;148:310-20.” in his post. This content was liked 2.1 million times. Considering the probability that this manuscript will be viewed 2.1 million times on an orthodontic journal page is very low, TikTok’s influence on the public can be understood.
Some of the content, which is defined as “irrelevant” and mostly videos with comedy, drama, and music, contained elements that threaten public health just to get likes from the audience. The best example of this was the video of a user named @thyrants, who is also an orthodontist, showed by sharing pictures of case examples that an influencer named @kylethomas who bonded his brackets to his teeth using non-orthodontic glue instead of orthodontic adhesive at home and then attach chain elastic to himself, which could lead to many types of dangerous complications. In this video, the risk that an influencer with 27 million followers could mislead the community and harm public health cannot be ruled out. Moreover, the original content dated February 13, 2021, on the user @kylethomas’s page received 1.5 million likes, was shared over 4000, and the video received 4000 comments. Torofdar reported a similar situation about a trend on TikTok in which users nailed the cutting edges of their front teeth to obtain a more straight smile. And they underlined that; that was an irreversible situation that has many potential risks that can damage the health of both streamers and viewers. The account which is also an orthodontist named @shineorthodontics uploaded a video about brushing. He also suggested brushing with baking soda on colored esthetic brackets. It received 49,900 likes.
It was seen that some users’ Instagram accounts were linked to their TikTok profiles and they were used to reach their Instagram accounts. Furthermore, the World Health Organization did have a TikTok account named @who with 2.9 million followers and 10 million likes. The orthodontics world should now see that the social attitudes of the patient, which is an indispensable part of the practice of this science as much as the doctor, should also be evaluated within the science of orthodontics. “Social orthodontics” should no longer be something that is ignored. It is not a scientific approach to turn a blind eye to platforms on which billions of posts are made in our profession. It should not be forgotten that the patient behavior patterns on TikTok, YouTube, Instagram, Facebook, and similar social media platforms are a sociological reality in an environment where the age is more and more digitized and tele-orthodontics is likely to set the agenda very shortly. Moreover, maybe it is time to create social media platforms that will function as a scientific YouTube, scientific TikTok that will be operated with peer-review processes.
Since information is now shared with more visual and auditory content and this type of sharing attracts more attention and acceptance; it seems that now it is necessary to develop a contemporary peer-review and evaluation system and scoring scales to be used to evaluate visual and auditory contents with the consensus of the orthodontic science world.
The limitation of this study is that not all videos uploaded on TikTok can be evaluated and that such studies are carried out on small sample pools. At the same time, the fact that the content uploaded to the application is constantly changing and the system is like organic living organism results in newly uploaded content being out of consideration. Considering this large and dynamic content of TikTok, this study can be considered as a pilot study that will lead to new studies since we will not have the chance to analyze all the content. It is a requirement for new studies to be about the phenomena of the changing age such as the internet of things (IoT), artificial intelligence (AI), deep learning, and tele-orthodontics.
Both quality and reliability of the content about orthodontics on TikTok were very low
The interest in TikTok videos, where billions of shares are made about orthodontics and are not subjected to scientific control, cannot be ignored. However, whether the content here is scientific or not is seriously open to debate
Even the most of the accounts were professionals, they did not prefer to give scientific information due to the mediums own dynamics and target population’s likes
TikTok, with a great audience population, is a very creative and entertaining method for sharing information with especially young people.
All authors made significant contributions to the design and production of the study and its writing. All authors reviewed the final draft of the article and approved it for submission.
Declaration of patient consent
Patient’s consent not required as there are no patients in this study.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
- Available from: https://www.tiktok.com [Last accessed on 2021 Jul 09]