Evaluating the agreement of users with usability problems identified by heuristic evaluation

•This paper Evaluated the agreement of users with usability problems.•The HE is the desired method for identifying recognizing, diagnosing problems.•The agreement of real user with problems identified by HE is at a medium level.•Younger users has a higher agreement with the problems identified in HE.•The users with lower computer experience had a higher agreement with the results of HE.•The severity of some problems identified by experts are not acknowledge by users. Heuristic evaluation (HE) is an inspection method that can be used for identifying usability problems of health information systems. There is insufficient evidence to what extent these problems are experienced by users in their working environment. The objective of this study was to investigate the agreement of real end users of a health information system with the problems identified by HE. First, the emergency information system of educational hospitals in Kerman, Iran was evaluated by a heuristic evaluation. Then, a valid and reliable (Cronbach’s alpha = 0.95) electronic questionnaire was designed based on the results of the first phase. Among 127 identified problems, the highest number of problems was related to the "consistency and standards" heuristic (18.9%). The user agreement coefficient with heuristic results was 0.5. The highest agreement of the participants was on the problems related to the "help user recognize, diagnose and recover from errors" heuristic (78%). There was a significant relationship between participants age and work experience with a computer (p