Ranking image search results using hover data

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for ranking images using hover data. In one aspect, a method includes determining a click count and a hover count for an image and a search query pair. The click count specifies a number of times that an image search result that includes a representation of the image has been selected when provided in response to the search query. The hover count specifies a number of times that the representation of the image has been hovered over when the image search result has been provided in response to the search query. A quality measure for the image with respect to the search query is determined. The quality measure is based on the click count and the hover count. A ranking of the image is adjusted for the search query based on the quality measure for the image.