Knowledge engineering using retrospective review of data: a useful technique or merely data dredging?

The process of extracting the knowledge or rules for medical decision making is not an easy task. One approach to knowledge engineering is to carefully review how decisions were made in the past with the goal of extracting the rules. The purpose of this project was to use previously collected data from ICU patients to derive the rules for the definition of hemodynamic stability. 97 ICU patients between 9/9/86 and 7/29/90 were included in the analysis. All of these patients had adult respiratory distress syndrome. Their mechanical ventilation was managed by a set of computerized protocols. We retrospectively searched the HELP system database for instructions that were not followed due to hemodynamic reasons. For each patient, we also chose one randomly selected therapy instruction which was followed to act as a control. For each instruction we then selected the corresponding hemodynamic data set. The data was then used in a stepwise logistic regression to determine the rules used for defining hemodynamic instability. We found that several of the hemodynamic parameters we had anticipated to be important were not even measured most of the time. The blood pressures and heart rate were almost identical between the hemodynamicly stable and unstable data sets. We conclude that the decision making process used by physicians has great variation, both between and within physicians. This makes knowledge engineering using retrospective techniques such as this prone to error and probably not very fruitful.