3226 DEVELOPMENT AND USABILITY ASSESSMENT OF AN AI-ENHANCED DASHBOARD SUPPORTING AVF MANAGEMENT IN CLINICAL PRACTICE

Abstract Background and Aims Despite the rise in publications reporting Artificial Intelligence (AI) solutions in healthcare, their usage in clinical practice remains limited. Prominent barriers for their implementation in clinical practice are the lack of integration of AI-based tools in the hospital health information system and clinical workflows. We previously developed an AVF Failure risk score to assess the expected incidence of AVF failures within 3 months based on routinely collected dialysis data. The risk score showed high accuracy (AUC = 0.81) and calibration. Based on this risk score, we developed an AI-enhanced dashboard supporting the management of AVF in hemodialysis patients. The usability and usefulness of the AI-enhanced AVF management dashboard was evaluated in a pilot quality improvement program. Method A quality improvement framework was used to evaluate the usability of an AI-enhanced dashboard for AVF management (Fig. 1A). First, healthcare professionals (HCP) were involved to provide inputs for model design, endpoint definition, and potential predictors through various qualitative interview techniques such as brainstorming sessions and focus groups (Phase I). A machine learning model was then developed to assess the expected incidence of AVF failure within 3 months. The model used routinely collected clinical data to minimize additional burden on clinical workflow. (Phase II). After developing and validating the model, a model error analysis was conducted using case studies methodology to improve model performance (Phase III). Once the model reached a good accuracy, think-aloud interview was used to abstract HCP needs and to design a user interface for model output. A user-friendly dashboard was developed to display the risk of AVF failure within 3-months and patient's vascular access history (Phase IV). This solution was integrated in the health record system and updated monthly with the latest risk score and medical information. A pilot quality improvement program was conducted in 15 clinics across 4 countries (IT, CZ, SK, SG) from May to December 2022. HCP were asked to evaluate their agreement with the risk score using a 5-point Likert scale (completely disagree to agree) and assess the usefulness of the dashboard in clinical decision making. At this stage the dashboard is not used for medical decision making (Phase V). The evaluation results were compiled into a monthly report and shared with all the HCP involved in the project