The value of Retrospective and Concurrent Think Aloud in formative usability testing of a physician data query tool

[Display omitted] •Comparison of Concurrent and Retrospective Think Aloud in usability testing of HIT.•Concurrent and Retrospective Think Aloud differ in usability problem detection scope.•Concurrent Think Aloud has higher effectiveness in terms of system redesign.•Retrospective Think Aloud is promoted for research on expert user customized design. To compare the performance of the Concurrent (CTA) and Retrospective (RTA) Think Aloud method and to assess their value in a formative usability evaluation of an Intensive Care Registry-physician data query tool designed to support ICU quality improvement processes. Sixteen representative intensive care physicians participated in the usability evaluation study. Subjects were allocated to either the CTA or RTA method by a matched randomized design. Each subject performed six usability-testing tasks of varying complexity in the query tool in a real-working context. Methods were compared with regard to number and type of problems detected. Verbal protocols of CTA and RTA were analyzed in depth to assess differences in verbal output. Standardized measures were applied to assess thoroughness in usability problem detection weighted per problem severity level and method overall effectiveness in detecting usability problems with regard to the time subjects spent per method. The usability evaluation of the data query tool revealed a total of 43 unique usability problems that the intensive care physicians encountered. CTA detected unique usability problems with regard to graphics/symbols, navigation issues, error messages, and the organization of information on the query tool’s screens. RTA detected unique issues concerning system match with subjects’ language and applied terminology. The in-depth verbal protocol analysis of CTA provided information on intensive care physicians’ query design strategies. Overall, CTA performed significantly better than RTA in detecting usability problems. CTA usability problem detection effectiveness was 0.80 vs. 0.62 (p