Performance evaluation of medical expert systems

The major problem in the evaluation of expert systems is the selection of the appropriate statistical measures of performance consistent with the parameters of the system domain. The objective of this paper is to develop the statistical evaluation methodology needed to assess the performance of medical expert systems including MEDAS — the Medical Emergency Decision Assistance System. The measures of performance are selected so as to have an operational interpretation and also reflect the predictive diagnostic capacity of a medical expert system. Certain summary measures are used that represent the sensitivity, specificity, and system response of a medical expert system. Measures of agreement such as the kappa statistic and the measure of conditional agrement are used to measure the agreement between the medical expert system and the physician. Goodman and Kruskal's lambda and tau measures of predictive association are introduced to evaluate the predictive capacity of a medical expert system. This methodology has been partially implemented in the performance evaluation of MEDAS.