Ranking Content Items Based on a Value of Learning

Methods, systems, and apparatus include computer programs encoded on a computer-readable storage medium, including a method for ranking content. A request for content is received. Eligible content items are identified, including a first eligible content item for which an uncertainty level of an associated expected click-through rate is above a predefined threshold. A subset of the eligible content items is evaluated, including the first eligible content item including producing a score. The score is a product of an associated bid and click-through rate for a given eligible content item. Producing the score includes adjusting a product of a bid times an expected click-through rate for the first eligible content item by a value of learning that represents a value for exploring the first eligible content item as a response to the request. The subset of eligible content items is ranked based on the produced scores.