Regression Testing Question Answering Cognitive Computing Systems by Applying Ground Truth Virtual Checksum Techniques

A mechanism is provided in a data processing system for performing regression testing on a question answering system instance. The mechanism trains a machine learning model for a question answering system using a ground truth virtual checksum as part of a ground truth including domain-specific ground truth. The ground truth virtual checksum comprises a set of test questions, an answer to each test question, and a confidence level range for each answer to a corresponding test question. The mechanism runs regression test buckets across system nodes with domain-specific corpora and receiving results from the system nodes. Each system node implements a question answering system instance of the question answering system by executing in accordance with the machine learning model and by accessing domain-specific corpora. Each test bucket includes a set of questions matching a subset of questions in the ground truth virtual checksum. The mechanism identifies regressions, inconsistencies, or destabilizations in code behavior in the system nodes based on results of comparing the results to the ground truth virtual checksum and generates a report presenting the identified regressions, inconsistencies, or destabilizations and the affected system nodes.