An Enumeration Problem in Ordered Sets Leads to Possible Benchmarks for Run-Time Prediction Algorithms

Motivated by the desire to estimate the number of order-preserving self maps of an ordered set, we compare three algorithms (Simple Sampling [4], Partial Backtracking [10] and Heuristic Sampling [1]) which predict how many nodes of a search tree are visited. The comparison is for the original algorithms that apply to backtracking and for modifications that apply to forward checking. We identify generic tree types and concrete, natural problems on which the algorithms predict incorrectly. We show that incorrect predictions not only occur because of large statistical variations but also because of (perceived) systemic biases of the prediction. Moreover, the quality of the prediction depends on the order of the variables. Our observations give new benchmarks for estimation and seem to make heuristic sampling the estimation algorithm of choice.