Estimating the number of faults: efficiency of removal, recapture, and seeding

Various experimental designs for estimating the number of faults in a system are studied including: (1) removal of each fault as it is detected; (2) marking of each fault as it is detected; and (3) introduction of a known number of faults into the system followed by (1) or (2). A unified framework is developed for comparing these designs; it also produces simplified estimators having high efficiency relative to maximum likelihood estimators. The designs are compared in terms of: (1) statistical accuracy; and (2) the number of failures that need to occur to achieve a given accuracy. On the basis of these comparisons, some general recommendations are made on the level of seeding as well as the choice of removal or recapture designs. When the testing effort is sufficient so that roughly two thirds of the faults are detected, the removal-design is preferred over the recapture-design, and there are no gains from seeding. However, this conclusion depends on assigning unit cost to all fault detections, which might not always be reasonable.