Simulation studies:do data generation techniques play favourites?

When comparing the performances of several classification procedures utilizing simulated data, it is necessary to ensure that the method of data generation does not unfairly favor one or more of the procedures. This study was undertaken to examine the influence of five methods of generation of data with dichotomous independent variables and an ordinal response. One generation method considered was model-free and four were model-based; it was hypothesized that the model-based generation techniques would unfairly favor their respective underlying models. In the three outcome category situation, it was found that the interaction between generation procedure and classification model, although statistically significant, was always very small. Thus, these results indicate that the choice of generation procedure may usually be made based on issues other than potential interaction between generation procedure and classification model.