BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectives
We present the Benchmark of Information Retrieval (IR) tasks with Complex Objectives (BIRCO). BIRCO evaluates the ability of IR systems to retrieve documents given multi-faceted user objectives. The benchmark's complexity and compact size make it suitable for evaluating large language model (LL...
Gespeichert in:
Veröffentlicht in: | arXiv.org 2024-04 |
---|---|
Hauptverfasser: | , , , , , |
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
container_end_page | |
---|---|
container_issue | |
container_start_page | |
container_title | arXiv.org |
container_volume | |
creator | Wang, Xiaoyue Wang, Jianyou Cao, Weili Wang, Kaicheng Paturi, Ramamohan Bergen, Leon |
description | We present the Benchmark of Information Retrieval (IR) tasks with Complex Objectives (BIRCO). BIRCO evaluates the ability of IR systems to retrieve documents given multi-faceted user objectives. The benchmark's complexity and compact size make it suitable for evaluating large language model (LLM)-based information retrieval systems. We present a modular framework for investigating factors that may influence LLM performance on retrieval tasks, and identify a simple baseline model which matches or outperforms existing approaches and more complex alternatives. No approach achieves satisfactory performance on all benchmark tasks, suggesting that stronger models and new retrieval protocols are necessary to address complex user needs. |
format | Article |
fullrecord | <record><control><sourceid>proquest</sourceid><recordid>TN_cdi_proquest_journals_2931010124</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2931010124</sourcerecordid><originalsourceid>FETCH-proquest_journals_29310101243</originalsourceid><addsrcrecordid>eNqNissKgkAUQIcgSMp_uNBa0BnttUspciWIe5nkiuNjxmZG6_Nz0QfEWZzFOSviUMYC7xRSuiGuMa3v-_RwpFHEHJLGaZ5kF7hCjLJqBq47UDWkslZ64FYoCTlaLXDmPRTcdAbewjaQqGHs8QPZs8XKihnNjqxr3ht0f96S_f1WJA9v1Oo1obFlqyYtl1TSMwv8BRqy_64v16I7Qg</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2931010124</pqid></control><display><type>article</type><title>BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectives</title><source>Free E- Journals</source><creator>Wang, Xiaoyue ; Wang, Jianyou ; Cao, Weili ; Wang, Kaicheng ; Paturi, Ramamohan ; Bergen, Leon</creator><creatorcontrib>Wang, Xiaoyue ; Wang, Jianyou ; Cao, Weili ; Wang, Kaicheng ; Paturi, Ramamohan ; Bergen, Leon</creatorcontrib><description>We present the Benchmark of Information Retrieval (IR) tasks with Complex Objectives (BIRCO). BIRCO evaluates the ability of IR systems to retrieve documents given multi-faceted user objectives. The benchmark's complexity and compact size make it suitable for evaluating large language model (LLM)-based information retrieval systems. We present a modular framework for investigating factors that may influence LLM performance on retrieval tasks, and identify a simple baseline model which matches or outperforms existing approaches and more complex alternatives. No approach achieves satisfactory performance on all benchmark tasks, suggesting that stronger models and new retrieval protocols are necessary to address complex user needs.</description><identifier>EISSN: 2331-8422</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Benchmarks ; Information retrieval ; Large language models ; Modular systems ; Task complexity</subject><ispartof>arXiv.org, 2024-04</ispartof><rights>2024. This work is published under http://creativecommons.org/licenses/by-sa/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>780,784</link.rule.ids></links><search><creatorcontrib>Wang, Xiaoyue</creatorcontrib><creatorcontrib>Wang, Jianyou</creatorcontrib><creatorcontrib>Cao, Weili</creatorcontrib><creatorcontrib>Wang, Kaicheng</creatorcontrib><creatorcontrib>Paturi, Ramamohan</creatorcontrib><creatorcontrib>Bergen, Leon</creatorcontrib><title>BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectives</title><title>arXiv.org</title><description>We present the Benchmark of Information Retrieval (IR) tasks with Complex Objectives (BIRCO). BIRCO evaluates the ability of IR systems to retrieve documents given multi-faceted user objectives. The benchmark's complexity and compact size make it suitable for evaluating large language model (LLM)-based information retrieval systems. We present a modular framework for investigating factors that may influence LLM performance on retrieval tasks, and identify a simple baseline model which matches or outperforms existing approaches and more complex alternatives. No approach achieves satisfactory performance on all benchmark tasks, suggesting that stronger models and new retrieval protocols are necessary to address complex user needs.</description><subject>Benchmarks</subject><subject>Information retrieval</subject><subject>Large language models</subject><subject>Modular systems</subject><subject>Task complexity</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>ABUWG</sourceid><sourceid>AFKRA</sourceid><sourceid>AZQEC</sourceid><sourceid>BENPR</sourceid><sourceid>CCPQU</sourceid><sourceid>DWQXO</sourceid><recordid>eNqNissKgkAUQIcgSMp_uNBa0BnttUspciWIe5nkiuNjxmZG6_Nz0QfEWZzFOSviUMYC7xRSuiGuMa3v-_RwpFHEHJLGaZ5kF7hCjLJqBq47UDWkslZ64FYoCTlaLXDmPRTcdAbewjaQqGHs8QPZs8XKihnNjqxr3ht0f96S_f1WJA9v1Oo1obFlqyYtl1TSMwv8BRqy_64v16I7Qg</recordid><startdate>20240403</startdate><enddate>20240403</enddate><creator>Wang, Xiaoyue</creator><creator>Wang, Jianyou</creator><creator>Cao, Weili</creator><creator>Wang, Kaicheng</creator><creator>Paturi, Ramamohan</creator><creator>Bergen, Leon</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope></search><sort><creationdate>20240403</creationdate><title>BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectives</title><author>Wang, Xiaoyue ; Wang, Jianyou ; Cao, Weili ; Wang, Kaicheng ; Paturi, Ramamohan ; Bergen, Leon</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-proquest_journals_29310101243</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Benchmarks</topic><topic>Information retrieval</topic><topic>Large language models</topic><topic>Modular systems</topic><topic>Task complexity</topic><toplevel>online_resources</toplevel><creatorcontrib>Wang, Xiaoyue</creatorcontrib><creatorcontrib>Wang, Jianyou</creatorcontrib><creatorcontrib>Cao, Weili</creatorcontrib><creatorcontrib>Wang, Kaicheng</creatorcontrib><creatorcontrib>Paturi, Ramamohan</creatorcontrib><creatorcontrib>Bergen, Leon</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science & Engineering Collection</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central UK/Ireland</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Publicly Available Content Database</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering Collection</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Wang, Xiaoyue</au><au>Wang, Jianyou</au><au>Cao, Weili</au><au>Wang, Kaicheng</au><au>Paturi, Ramamohan</au><au>Bergen, Leon</au><format>book</format><genre>document</genre><ristype>GEN</ristype><atitle>BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectives</atitle><jtitle>arXiv.org</jtitle><date>2024-04-03</date><risdate>2024</risdate><eissn>2331-8422</eissn><abstract>We present the Benchmark of Information Retrieval (IR) tasks with Complex Objectives (BIRCO). BIRCO evaluates the ability of IR systems to retrieve documents given multi-faceted user objectives. The benchmark's complexity and compact size make it suitable for evaluating large language model (LLM)-based information retrieval systems. We present a modular framework for investigating factors that may influence LLM performance on retrieval tasks, and identify a simple baseline model which matches or outperforms existing approaches and more complex alternatives. No approach achieves satisfactory performance on all benchmark tasks, suggesting that stronger models and new retrieval protocols are necessary to address complex user needs.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><oa>free_for_read</oa></addata></record> |
fulltext | fulltext |
identifier | EISSN: 2331-8422 |
ispartof | arXiv.org, 2024-04 |
issn | 2331-8422 |
language | eng |
recordid | cdi_proquest_journals_2931010124 |
source | Free E- Journals |
subjects | Benchmarks Information retrieval Large language models Modular systems Task complexity |
title | BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectives |
url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-24T05%3A57%3A47IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=document&rft.atitle=BIRCO:%20A%20Benchmark%20of%20Information%20Retrieval%20Tasks%20with%20Complex%20Objectives&rft.jtitle=arXiv.org&rft.au=Wang,%20Xiaoyue&rft.date=2024-04-03&rft.eissn=2331-8422&rft_id=info:doi/&rft_dat=%3Cproquest%3E2931010124%3C/proquest%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=2931010124&rft_id=info:pmid/&rfr_iscdi=true |