Evaluating performance of binary classification systems

Methods and apparatus are described for evaluating a binary classification system operable to classify each of a plurality of events as a first event type or a second event type. At least some of the events of the first event type are independently verifiable with reference to verification data. The binary classification system is susceptible to a first error type in which events of the first event type are classified as the second event type, and a second error type in which events of the second event type are classified as the first event type. Operation of a first configuration of the binary classification system is evaluated with reference to an objective function. The objective function is derived by expressing a number of errors of the second error type in terms of a number of errors of the first error type with reference to the verification data, and by assuming relative proportions of the first and second event types within the plurality of events.