Patients’ involvement in e-health services quality assessment: A system for the automatic interpretation of SMS-based patients’ feedback

•We use patients’ perception of an outpatients reminder system to improve the service itself.•We extract such perceptions from their SMS responses to the reminder.•We automatically access the content of SMS messages.•We follow a machine learning based approach using CRFs.•The adoption of patients’ feedback show a positive impact on service functioning. Effective communication between patients and health services providers is a key aspect for optimizing and maintaining these services. This work describes a system for the automatic evaluation of users’ perception of the quality of SmsCup, a reminder system for outpatient visits based on short message service (SMS). The final purpose is the creation of a closed-loop control system for the outpatient service, where patients’ complaints and comments represent a feedback that can be used for a better implementation of the service itself. SmsCup was adopted since about eight years by an Italian healthcare organization, with very good results in reducing the no-show (missing visits) phenomenon. During these years, a number of citizens, even if not required, sent a message back, with comments about the service. The automatic interpretation of the content of those SMS may be useful for monitoring and improving service performances.Yet, due to the complex nature of SMS language, their interpretation represents an ongoing challenge. The proposed system uses conditional random fields as the information extraction method for classifying messages into several semantic categories. The categories refer to appreciation of the service or complaints of various types. Then, the system analyzes the extracted content and provides feedback to the service providers, making them learning and acting on this basis. At each step, the content of the messages reveals the actual state of the service as well as the efficacy of corrective actions previously undertaken. Our evaluations showed that: (i) the SMS classification system has achieved good overall performance with an average F1-measure and an overall accuracy of about 92%; (ii) the notification of the patients’ feedbacks to service providers showed a positive impact on service functioning. Our study proposed an interactive patient-centered system for continuous monitoring of the service quality. It has demonstrated the feasibility of a tool for the analysis and notification of the patients’ feedback on their service experiences, which would support a more regular access to the ser