Evidence Contextualization and Counterfactual Attribution for Conversational QA over Heterogeneous Data with RAG Systems

Retrieval Augmented Generation (RAG) works as a backbone for interacting with an enterprise's own data via Conversational Question Answering (ConvQA). In a RAG system, a retriever fetches passages from a collection in response to a question, which are then included in the prompt of a large lang...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:arXiv.org 2024-12
Hauptverfasser: Rishiraj Saha Roy, Schlotthauer, Joel, Hinze, Chris, Foltyn, Andreas, Hahn, Luzian, Kuech, Fabian
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page
container_issue
container_start_page
container_title arXiv.org
container_volume
creator Rishiraj Saha Roy
Schlotthauer, Joel
Hinze, Chris
Foltyn, Andreas
Hahn, Luzian
Kuech, Fabian
description Retrieval Augmented Generation (RAG) works as a backbone for interacting with an enterprise's own data via Conversational Question Answering (ConvQA). In a RAG system, a retriever fetches passages from a collection in response to a question, which are then included in the prompt of a large language model (LLM) for generating a natural language (NL) answer. However, several RAG systems today suffer from two shortcomings: (i) retrieved passages usually contain their raw text and lack appropriate document context, negatively impacting both retrieval and answering quality; and (ii) attribution strategies that explain answer generation usually rely only on similarity between the answer and the retrieved passages, thereby only generating plausible but not causal explanations. In this work, we demonstrate RAGONITE, a RAG system that remedies the above concerns by: (i) contextualizing evidence with source metadata and surrounding text; and (ii) computing counterfactual attribution, a causal explanation approach where the contribution of an evidence to an answer is determined by the similarity of the original response to the answer obtained by removing that evidence. To evaluate our proposals, we release a new benchmark ConfQuestions, with 300 hand-created conversational questions, each in English and German, coupled with ground truth URLs, completed questions, and answers from 215 public Confluence pages, that are typical of enterprise wiki spaces with heterogeneous elements. Experiments with RAGONITE on ConfQuestions show the viability of our ideas: contextualization improves RAG performance, and counterfactual attribution is effective at explaining RAG answers.
format Article
fullrecord <record><control><sourceid>proquest</sourceid><recordid>TN_cdi_proquest_journals_3145904974</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>3145904974</sourcerecordid><originalsourceid>FETCH-proquest_journals_31459049743</originalsourceid><addsrcrecordid>eNqNjMkKAjEQRIMgKOo_NHgWxmTG5Ti4Xl3uQ9QejWii6Y7b1xvFD_BUVL2qqoi6VKrbGaRS1kSL6Jgkiez1ZZapunhMbmaHdoswcpbxwUGfzEuzcRa03cU0xNiXevshkDN7swlfXDr_Gd3Q07cf8SIHFz3MMW7cHi26QDDWrOFu-ADLfAarJzGeqSmqpT4Rtn7aEO3pZD2ady7eXQMSF0cXfDylQnXTbJikw36q_mu9AQ-2Tos</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>3145904974</pqid></control><display><type>article</type><title>Evidence Contextualization and Counterfactual Attribution for Conversational QA over Heterogeneous Data with RAG Systems</title><source>Free E- Journals</source><creator>Rishiraj Saha Roy ; Schlotthauer, Joel ; Hinze, Chris ; Foltyn, Andreas ; Hahn, Luzian ; Kuech, Fabian</creator><creatorcontrib>Rishiraj Saha Roy ; Schlotthauer, Joel ; Hinze, Chris ; Foltyn, Andreas ; Hahn, Luzian ; Kuech, Fabian</creatorcontrib><description>Retrieval Augmented Generation (RAG) works as a backbone for interacting with an enterprise's own data via Conversational Question Answering (ConvQA). In a RAG system, a retriever fetches passages from a collection in response to a question, which are then included in the prompt of a large language model (LLM) for generating a natural language (NL) answer. However, several RAG systems today suffer from two shortcomings: (i) retrieved passages usually contain their raw text and lack appropriate document context, negatively impacting both retrieval and answering quality; and (ii) attribution strategies that explain answer generation usually rely only on similarity between the answer and the retrieved passages, thereby only generating plausible but not causal explanations. In this work, we demonstrate RAGONITE, a RAG system that remedies the above concerns by: (i) contextualizing evidence with source metadata and surrounding text; and (ii) computing counterfactual attribution, a causal explanation approach where the contribution of an evidence to an answer is determined by the similarity of the original response to the answer obtained by removing that evidence. To evaluate our proposals, we release a new benchmark ConfQuestions, with 300 hand-created conversational questions, each in English and German, coupled with ground truth URLs, completed questions, and answers from 215 public Confluence pages, that are typical of enterprise wiki spaces with heterogeneous elements. Experiments with RAGONITE on ConfQuestions show the viability of our ideas: contextualization improves RAG performance, and counterfactual attribution is effective at explaining RAG answers.</description><identifier>EISSN: 2331-8422</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Large language models ; Questions ; Retrieval ; Similarity</subject><ispartof>arXiv.org, 2024-12</ispartof><rights>2024. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>780,784</link.rule.ids></links><search><creatorcontrib>Rishiraj Saha Roy</creatorcontrib><creatorcontrib>Schlotthauer, Joel</creatorcontrib><creatorcontrib>Hinze, Chris</creatorcontrib><creatorcontrib>Foltyn, Andreas</creatorcontrib><creatorcontrib>Hahn, Luzian</creatorcontrib><creatorcontrib>Kuech, Fabian</creatorcontrib><title>Evidence Contextualization and Counterfactual Attribution for Conversational QA over Heterogeneous Data with RAG Systems</title><title>arXiv.org</title><description>Retrieval Augmented Generation (RAG) works as a backbone for interacting with an enterprise's own data via Conversational Question Answering (ConvQA). In a RAG system, a retriever fetches passages from a collection in response to a question, which are then included in the prompt of a large language model (LLM) for generating a natural language (NL) answer. However, several RAG systems today suffer from two shortcomings: (i) retrieved passages usually contain their raw text and lack appropriate document context, negatively impacting both retrieval and answering quality; and (ii) attribution strategies that explain answer generation usually rely only on similarity between the answer and the retrieved passages, thereby only generating plausible but not causal explanations. In this work, we demonstrate RAGONITE, a RAG system that remedies the above concerns by: (i) contextualizing evidence with source metadata and surrounding text; and (ii) computing counterfactual attribution, a causal explanation approach where the contribution of an evidence to an answer is determined by the similarity of the original response to the answer obtained by removing that evidence. To evaluate our proposals, we release a new benchmark ConfQuestions, with 300 hand-created conversational questions, each in English and German, coupled with ground truth URLs, completed questions, and answers from 215 public Confluence pages, that are typical of enterprise wiki spaces with heterogeneous elements. Experiments with RAGONITE on ConfQuestions show the viability of our ideas: contextualization improves RAG performance, and counterfactual attribution is effective at explaining RAG answers.</description><subject>Large language models</subject><subject>Questions</subject><subject>Retrieval</subject><subject>Similarity</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2024</creationdate><recordtype>article</recordtype><sourceid>ABUWG</sourceid><sourceid>AFKRA</sourceid><sourceid>AZQEC</sourceid><sourceid>BENPR</sourceid><sourceid>CCPQU</sourceid><sourceid>DWQXO</sourceid><recordid>eNqNjMkKAjEQRIMgKOo_NHgWxmTG5Ti4Xl3uQ9QejWii6Y7b1xvFD_BUVL2qqoi6VKrbGaRS1kSL6Jgkiez1ZZapunhMbmaHdoswcpbxwUGfzEuzcRa03cU0xNiXevshkDN7swlfXDr_Gd3Q07cf8SIHFz3MMW7cHi26QDDWrOFu-ADLfAarJzGeqSmqpT4Rtn7aEO3pZD2ady7eXQMSF0cXfDylQnXTbJikw36q_mu9AQ-2Tos</recordid><startdate>20241213</startdate><enddate>20241213</enddate><creator>Rishiraj Saha Roy</creator><creator>Schlotthauer, Joel</creator><creator>Hinze, Chris</creator><creator>Foltyn, Andreas</creator><creator>Hahn, Luzian</creator><creator>Kuech, Fabian</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope></search><sort><creationdate>20241213</creationdate><title>Evidence Contextualization and Counterfactual Attribution for Conversational QA over Heterogeneous Data with RAG Systems</title><author>Rishiraj Saha Roy ; Schlotthauer, Joel ; Hinze, Chris ; Foltyn, Andreas ; Hahn, Luzian ; Kuech, Fabian</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-proquest_journals_31459049743</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2024</creationdate><topic>Large language models</topic><topic>Questions</topic><topic>Retrieval</topic><topic>Similarity</topic><toplevel>online_resources</toplevel><creatorcontrib>Rishiraj Saha Roy</creatorcontrib><creatorcontrib>Schlotthauer, Joel</creatorcontrib><creatorcontrib>Hinze, Chris</creatorcontrib><creatorcontrib>Foltyn, Andreas</creatorcontrib><creatorcontrib>Hahn, Luzian</creatorcontrib><creatorcontrib>Kuech, Fabian</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science &amp; Engineering Collection</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central UK/Ireland</collection><collection>ProQuest Central Essentials</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Access via ProQuest (Open Access)</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering Collection</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Rishiraj Saha Roy</au><au>Schlotthauer, Joel</au><au>Hinze, Chris</au><au>Foltyn, Andreas</au><au>Hahn, Luzian</au><au>Kuech, Fabian</au><format>book</format><genre>document</genre><ristype>GEN</ristype><atitle>Evidence Contextualization and Counterfactual Attribution for Conversational QA over Heterogeneous Data with RAG Systems</atitle><jtitle>arXiv.org</jtitle><date>2024-12-13</date><risdate>2024</risdate><eissn>2331-8422</eissn><abstract>Retrieval Augmented Generation (RAG) works as a backbone for interacting with an enterprise's own data via Conversational Question Answering (ConvQA). In a RAG system, a retriever fetches passages from a collection in response to a question, which are then included in the prompt of a large language model (LLM) for generating a natural language (NL) answer. However, several RAG systems today suffer from two shortcomings: (i) retrieved passages usually contain their raw text and lack appropriate document context, negatively impacting both retrieval and answering quality; and (ii) attribution strategies that explain answer generation usually rely only on similarity between the answer and the retrieved passages, thereby only generating plausible but not causal explanations. In this work, we demonstrate RAGONITE, a RAG system that remedies the above concerns by: (i) contextualizing evidence with source metadata and surrounding text; and (ii) computing counterfactual attribution, a causal explanation approach where the contribution of an evidence to an answer is determined by the similarity of the original response to the answer obtained by removing that evidence. To evaluate our proposals, we release a new benchmark ConfQuestions, with 300 hand-created conversational questions, each in English and German, coupled with ground truth URLs, completed questions, and answers from 215 public Confluence pages, that are typical of enterprise wiki spaces with heterogeneous elements. Experiments with RAGONITE on ConfQuestions show the viability of our ideas: contextualization improves RAG performance, and counterfactual attribution is effective at explaining RAG answers.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier EISSN: 2331-8422
ispartof arXiv.org, 2024-12
issn 2331-8422
language eng
recordid cdi_proquest_journals_3145904974
source Free E- Journals
subjects Large language models
Questions
Retrieval
Similarity
title Evidence Contextualization and Counterfactual Attribution for Conversational QA over Heterogeneous Data with RAG Systems
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-25T03%3A42%3A08IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=document&rft.atitle=Evidence%20Contextualization%20and%20Counterfactual%20Attribution%20for%20Conversational%20QA%20over%20Heterogeneous%20Data%20with%20RAG%20Systems&rft.jtitle=arXiv.org&rft.au=Rishiraj%20Saha%20Roy&rft.date=2024-12-13&rft.eissn=2331-8422&rft_id=info:doi/&rft_dat=%3Cproquest%3E3145904974%3C/proquest%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=3145904974&rft_id=info:pmid/&rfr_iscdi=true