BEYOND – Building an Experience to Yield Optimal Nautical Displays – Measuring Nautical Situation Awareness

The growing digitisation and integration of ship operations, navigation, communication, safety equipment and systems have expanded the functional scope of modern navigation systems. However, this has also increased the workload for the crew, making it more challenging to cultivate situation awareness and make appropriate decisions, particularly in critical situations. The research project, BEYOND – Building an Experience to Yield Optimal Nautical Displays, aims to develop and validate a test methodology for evaluating maritime Human-Machine Interfaces (HMI). Besides system usability and user experience, maritime HMIs are generally evaluated in terms of the situation awareness acquired during navigation. For this purpose, the established Situation Awareness Global Assessment Technique (SAGAT) was adapted such that participants experienced a series of simulated navigation scenarios and were interrupted at unexpected intervals to answer situation awareness questions. This paper addresses the challenge of formulating the appropriate situation awareness questions and their associated frameworks. An assembly of questions reflecting the knowledge of a navigator was created, taking into account different categories of information, including the own ship, the route taken, and the surrounding area. The number of questions needed to capture the necessary knowledge comprehensively was also defined. Additionally, it was crucial to determine the appropriate answer options for each question. Besides graphically presented multiple-choice questions, where the answer had to be selected based on a screenshot of the environment, there are also alphanumeric questions where the participant had to retrieve knowledge from the situation and enter numerical values. For instance, this included information about the course, navigation hazards, etc. A major challenge in assessing these numerical values was determining the acceptable range, i.e., the thresholds that define an answer as correct or incorrect. This paper outlines the BEYOND methodology developed for the comprehensive testing of maritime HMIs, focusing primarily on situation awareness assessment. It describes the steps taken to compile a list of situation awareness questions and their corresponding accuracy thresholds based on the findings from expert interviews. The paper also discusses the experimental results of a user study conducted to test the overall methodology.