Psychiatry Transcript Annotation: Process Study and Improvements

Background: The demand for psychiatry is increasing each year. Limited research has been performed to improve psychiatrist work experience and reduce daily workload using computational methods. There is currently no validated tool or procedure for the mental health transcript annotation process for generating “gold-standard” data. The purpose of this paper was to determine the annotation process for mental health transcripts and how it can be improved to acquire more reliable results considering human factors elements. Method: Three expert clinicians were recruited in this study to evaluate the transcripts. The clinicians were asked to fully annotate two transcripts. An additional five subjects were recruited randomly (aged between 20-40) for this pilot study, which was divided into two phases, phase 1 (annotation without training) and phase 2 (annotation with training) of five transcripts. Kappa statistics were used to measure the inter-rater reliability and accuracy between subjects. Results: The inter-rater reliability between expert clinicians for two transcripts were 0.26 (CI 0.19 to 0.33) and 0.49 (CI 0.42 to 0.57), respectively. In the pilot testing phases, the mean inter-rater reliability between subjects was higher in phase 2 with training transcript (k= 0.35 (CI 0.052 to 0.625)) than in phase 1 without training transcript (k= 0.29 (CI 0.128 to 0.451)). After training, the accuracy percentage among subjects was significantly higher in transcript A (p=0.04) than transcript B (p=0.10). Conclusion: This study focused on understanding the annotation process for mental health transcripts, which will be applied in training machine learning models. Through this exploratory study, the research found appropriate categorical labels that should be included for transcripts annotation, and the importance of training the subjects. Contributions of this case study will help the psychiatric clinicians and researchers in implementing the recommended data collection process to develop a more accurate artificial intelligence model for fully- or semi-automated transcript annotation.