Quality metrics for search engine deterministic sort orders

eCommerce search engines such as eBay and Amazon often allow the user to order their search results on deterministic features such as price, time, or distance from the seller. Using metrics such as precision at 10 documents (P@10), others have already shown that the quality of deterministically sorted results is lower than that of best-match (or relevance) sorted results, and that work is needed in order to improve result quality. But metrics such as P@10 are based purely on relevance, and do not reflect the order-feature: cost (be it price, time, or otherwise) — and it is hard to see how to improve a system without a metric that reflects the quality of the ordering. In this contribution we introduce a set of metrics that, using relevance and cost, measure the quality of deterministically sorted search engine results. We examine metrics from the perspective of the buyer, the seller, and the systems engineer. Using our new metrics (buying power (bp), buying power for K (bp4k), selling power (sp), and cheapest precision (Pc)) we re-evaluate the results of the “eBay SIGIR 2019 eCommerce Search Challenge: High Accuracy Recall Task” and demonstrate how to include cost in a metric designed to evaluate the quality of deterministically ordered lists of results. •Metrics to measure the quality of search engine deterministic sort orderings.•Search engine metrics including both a relevance and cost component.•Relevance from the perspective of the buyer, the seller, and the systems engineer.•Multiple user models with well-grounded metrics for each.