Exploring the Nexus Between Retrievability and Query Generation Strategies
Quantifying bias in retrieval functions through document retrievability scores is vital for assessing recall-oriented retrieval systems. However, many studies investigating retrieval model bias lack validation of their query generation methods as accurate representations of retrievability for real u...
Gespeichert in:
Hauptverfasser: | , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | Quantifying bias in retrieval functions through document retrievability
scores is vital for assessing recall-oriented retrieval systems. However, many
studies investigating retrieval model bias lack validation of their query
generation methods as accurate representations of retrievability for real users
and their queries. This limitation results from the absence of established
criteria for query generation in retrievability assessments. Typically,
researchers resort to using frequent collocations from document corpora when no
query log is available. In this study, we address the issue of reproducibility
and seek to validate query generation methods by comparing retrievability
scores generated from artificially generated queries to those derived from
query logs. Our findings demonstrate a minimal or negligible correlation
between retrievability scores from artificial queries and those from query
logs. This suggests that artificially generated queries may not accurately
reflect retrievability scores as derived from query logs. We further explore
alternative query generation techniques, uncovering a variation that exhibits
the highest correlation. This alternative approach holds promise for improving
reproducibility when query logs are unavailable. |
---|---|
DOI: | 10.48550/arxiv.2404.09473 |