EXPLAINING SEMANTIC SEARCH

The invention uses document retrieval to explain to a human user the properties of a query object that are revealed by a machine learning procedure, lending interpretability to the procedure. A query object is compared to reference objects by transforming the query object and reference objects into representative tokens. Reference objects with many tokens in common with the query object are returned as relevant result objects by a document retrieval system. At this stage, a human user can observe commonalities and differences between the result objects and query object to understand which query object properties are captured by the query object's token representation. In one embodiment, the token transformation is the composition of an embedding mapping trained by machine learning with a locality sensitive hash function. In a further embodiment, if the reference objects are labeled, the result objects can be grouped according to their labels; the labels of groups with many result objects relevant to the query object are suggested labels for the query object, and a human user can compare the query object to the result objects in each group to infer the reason for each suggested label. In a further embodiment, we incorporate differential highlighting. Differential highlighting combines the sensitivities of the query object's tokens to each token from each result group. The resulting overall sensitivity of the query object to each group highlights features of the query object pertinent to each group label. In one embodiment, the interest objects are 2-dimensional or 3-dimensional images, and the highlights are regions of the query image that most contribute to each label suggested by result groupings. In another embodiment, the interest objects are audio clips. In another embodiment, the interest objects are blocks of text.