Using machine learning to analyse large volume of public health data to drive service improvement

Background The UK government's approach to the pandemic relies on a test, trace and isolate strategy, mainly implemented via the digital Contact Tracing and Advice Service (CTAS). Feedback on user experience is central to the successful development of public-facing services. As the situation dynamically changes and data accumulate, interpretation of feedback by humans becomes time-consuming and unreliable. The aim was to evaluate the use of Machine Learning (ML) techniques as tools to understand the issues with the Service as expressed in the free-text responses of the users. The specific objectives were to 1) conduct an analysis of supervised and unsupervised techniques to develop the most optimal model, 2) generate actionable themes that can be used to increase user satisfaction of the CTAS. Methods We evaluated and compared 5 supervised classification algorithms in terms of serviceability and accuracy. We proceeded by evaluating an unsupervised Topic Modelling approach, testing models with 5-40 topics and differing covariates in terms of coherence, residuals and interpretability by human coders. Two human coders conducted thematic analysis to interpret the topics. Results Due to the low accuracy, the degree of human involvement and broadness of themes we found that a supervised ML approach was not well suited to our objective. We identified a Structural Topic Model with 25 topics and metadata as covariates as the most appropriate for acquiring insights. Preliminary results from analysis of the feedback by 16,262 users from May 2020 to March 2021 highlighted issues with the Service falling within three major themes: lack of data coordination, ineffective communication, and technical issues. Conclusions Structural Topic Modelling was found to be the most effective technique to rapidly acquire user insights. The 25 topics provided highly specific insights of issues that can be utilized towards improving the CTAS. Key messages A ML approach can be a quick and cost-effective method to provide high quality, actionable insights from free-text feedback in order to optimize public health services. Topic models can rapidly provide highly specific user insights with minimal human involvement and low maintenance requirements, making them ideal evaluation tools for pandemic response services.