Validation of a knowledge based reminder system for diagnostic test ordering in general practice

We describe the validation of a real-time automated reminder system that assists General Practitioners (GP) in appropriate test ordering. We compared the comments of human experts with the comments of the reminder system using a retrospective random selection of 253 request forms. A panel of three expert physicians judged the requested tests independently based on their interpretations of the practice guidelines. The majority assessment of the physicians was compared with the assessment of the reminder system. In case the system's output differed from the majority assessment the written practice guidelines were consulted. On average 1.75 reminders were produced per form. In total 32 of the 442 given reminders (7%) were given incorrectly. The amount of information and the level of detail (the specificity of the terms) in which the GP describes the patients' medical status are crucial for the reminder system to react correctly.