Training software engineers in a novel usability evaluation technique

Novel approaches to designing or analysing systems only become useful when they are usable by practitioners in the field, and not just by their originators. Design techniques often fail to make the transition from research to practice because insufficient attention is paid to understanding and communicating the skills required to use them. This paper reports on work to train software engineering students to use a user-centred language for describing and analysing interface designs called the “Programmable User Model Instruction Language”, or IL. Various types of data, including video, students' IL descriptions and brief usability reports were collected during training, and subsequently analysed. These show that after 6 h of training, students have a good grasp of the syntax of the notation, and start using notational affordances to support their reasoning, but that their reasoning is still limited by a poor grasp of the underlying cognitive theory. A comparison of the analyses of trainees with those of experts provides a means of developing a better understanding of the nature of expertise in this area—as comprising an understanding of the syntax and the surface semantics of the notation, the underlying ognitive theory, the method of conducting an analysis and the implications of the analysis for design.