Can an Algorithm Tell How Spiritual You Are? Using Generative Pretrained Transformers for Sophisticated Forms of Text Analysis

Text analysis is a form of psychological assessment that involves converting qualitative information (text) into quantitative data. We tested whether automated text analysis using Generative Pre-trained Transformers (GPTs) can match the "gold standard" of manual text analysis, even when assessing a highly nuanced construct like spirituality. In Study 1, N = 2199 US undergraduates wrote about their goals (N = 6597 texts) and completed self-reports of spirituality and theoretically related constructs (religiousness and mental health). In Study 2, N = 357 community adults wrote short essays (N = 714 texts) and completed trait self-reports, 5 weeks of daily diaries, and behavioral measures of spirituality. Trained research assistants and GPTs then coded the texts for spirituality. The GPTs performed just as well as human raters. Human- and GPT-generated scores were remarkably consistent and showed equivalent associations with other measures of spirituality and theoretically related constructs. GPTs can match the gold standard set by human raters, even in sophisticated forms of text analysis, but require a fraction of the time and labor.