The Use of Interobserver Methods in System Tests: A Methodological Case Study

An interobserver approach to conducting a system test is presented in a case study to promote this methodology for human factors practitioners. A system test of a specific feature of a commercially available VCR was performed under a contract. This special feature was designed to automatically fast-forward through recorded commercials during playback. Two specific measures of system performance required under this contract were the accuracy in skipping commercials and the accuracy in not interfering with regular programs during playback. In order to obtain a high level of reliability and validity at a relative low cost, an interobserver methodology was used to measure the accuracy of the VCR commercial skip feature. This method entailed having pairs of observers from a group of ten observers analyze 52 videotapes for commercial and program errors, comparing and reconciling their data, assessing interobserver reliability, and using the means of their observations to calculate accuracy rates. The benefits of multiple observer accuracy measures are discussed.