Evaluation of diagnostic classifiers using artificial clinical cases

Evaluation of classifiers in diagnosis support systems is a non-trivial task. It can be done in a form of controlled and blinded clinical trial, which is often difficult and costly. We propose a new method for generating artificial medical cases from a knowledge base, utilizing the concept of so-called medical diamonds. Cases generated using this method have features analogous to that of double-blinded trial and, thus, can be used for measuring sensitivity and specificity of diagnostic classifiers. This is easy and low-cost method of evaluation and comparison of classifiers in diagnosis support systems. We demonstrate that this method is able to produce valuable results when used for evaluation of similarity-based classifiers as well as shallow and deep neural networks.