Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading

Large language models (LLMs) have advanced in large strides due to the effectiveness of the self-attention mechanism that processes and compares all tokens at once. However, this mechanism comes with a fundamental issue -- the predetermined context window is bound to be limited. Despite attempts to...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: Chen, Howard, Pasunuru, Ramakanth, Weston, Jason, Celikyilmaz, Asli
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page
container_issue
container_start_page
container_title
container_volume
creator Chen, Howard
Pasunuru, Ramakanth
Weston, Jason
Celikyilmaz, Asli
description Large language models (LLMs) have advanced in large strides due to the effectiveness of the self-attention mechanism that processes and compares all tokens at once. However, this mechanism comes with a fundamental issue -- the predetermined context window is bound to be limited. Despite attempts to extend the context window through methods like extrapolating the positional embedding, using recurrence, or selectively retrieving essential parts of the long sequence, long-text understanding continues to be a challenge. We propose an alternative approach which instead treats the LLM as an interactive agent, allowing it to decide how to read the text via iterative prompting. We introduce MemWalker, a method that first processes the long context into a tree of summary nodes. Upon receiving a query, the model navigates this tree in search of relevant information, and responds once it gathers sufficient information. On long-text question answering tasks our method outperforms baseline approaches that use long context windows, recurrence, and retrieval. We show that, beyond effective reading, MemWalker enhances explainability by highlighting the reasoning steps as it interactively reads the text; pinpointing the relevant text segments related to the query.
doi_str_mv 10.48550/arxiv.2310.05029
format Article
fullrecord <record><control><sourceid>arxiv_GOX</sourceid><recordid>TN_cdi_arxiv_primary_2310_05029</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2310_05029</sourcerecordid><originalsourceid>FETCH-LOGICAL-a679-1a1a498ecaa91eda7ddf07176ad9a3855e14427e65b95bc40f0940ddde0f680e3</originalsourceid><addsrcrecordid>eNotj8tOwzAURL1hgQofwKr-gZTrxI5jdhBelVIqoUoso9v4prVoEmRMafj6msJqpNHoaA5jVwJmslAKrtEf3H6WZrEABak5Zy9vuHt3_YbfD989D1viC-oGP_IF_tANv6Nx6C0vhz7QIfDKdS7ElR--Nls-j6XHJrg98VdCGzEX7KzF3Sdd_ueErR4fVuVzUi2f5uVtlWCuTSJQoDQFNYhGkEVtbQta6BytwSw-JSFlqilXa6PWjYQWjARrLUGbF0DZhE3_sCeh-sO7Dv1Y_4rVJ7HsCHI-SPo</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading</title><source>arXiv.org</source><creator>Chen, Howard ; Pasunuru, Ramakanth ; Weston, Jason ; Celikyilmaz, Asli</creator><creatorcontrib>Chen, Howard ; Pasunuru, Ramakanth ; Weston, Jason ; Celikyilmaz, Asli</creatorcontrib><description>Large language models (LLMs) have advanced in large strides due to the effectiveness of the self-attention mechanism that processes and compares all tokens at once. However, this mechanism comes with a fundamental issue -- the predetermined context window is bound to be limited. Despite attempts to extend the context window through methods like extrapolating the positional embedding, using recurrence, or selectively retrieving essential parts of the long sequence, long-text understanding continues to be a challenge. We propose an alternative approach which instead treats the LLM as an interactive agent, allowing it to decide how to read the text via iterative prompting. We introduce MemWalker, a method that first processes the long context into a tree of summary nodes. Upon receiving a query, the model navigates this tree in search of relevant information, and responds once it gathers sufficient information. On long-text question answering tasks our method outperforms baseline approaches that use long context windows, recurrence, and retrieval. We show that, beyond effective reading, MemWalker enhances explainability by highlighting the reasoning steps as it interactively reads the text; pinpointing the relevant text segments related to the query.</description><identifier>DOI: 10.48550/arxiv.2310.05029</identifier><language>eng</language><subject>Computer Science - Computation and Language</subject><creationdate>2023-10</creationdate><rights>http://arxiv.org/licenses/nonexclusive-distrib/1.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,780,885</link.rule.ids><linktorsrc>$$Uhttps://arxiv.org/abs/2310.05029$$EView_record_in_Cornell_University$$FView_record_in_$$GCornell_University$$Hfree_for_read</linktorsrc><backlink>$$Uhttps://doi.org/10.48550/arXiv.2310.05029$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Chen, Howard</creatorcontrib><creatorcontrib>Pasunuru, Ramakanth</creatorcontrib><creatorcontrib>Weston, Jason</creatorcontrib><creatorcontrib>Celikyilmaz, Asli</creatorcontrib><title>Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading</title><description>Large language models (LLMs) have advanced in large strides due to the effectiveness of the self-attention mechanism that processes and compares all tokens at once. However, this mechanism comes with a fundamental issue -- the predetermined context window is bound to be limited. Despite attempts to extend the context window through methods like extrapolating the positional embedding, using recurrence, or selectively retrieving essential parts of the long sequence, long-text understanding continues to be a challenge. We propose an alternative approach which instead treats the LLM as an interactive agent, allowing it to decide how to read the text via iterative prompting. We introduce MemWalker, a method that first processes the long context into a tree of summary nodes. Upon receiving a query, the model navigates this tree in search of relevant information, and responds once it gathers sufficient information. On long-text question answering tasks our method outperforms baseline approaches that use long context windows, recurrence, and retrieval. We show that, beyond effective reading, MemWalker enhances explainability by highlighting the reasoning steps as it interactively reads the text; pinpointing the relevant text segments related to the query.</description><subject>Computer Science - Computation and Language</subject><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><sourceid>GOX</sourceid><recordid>eNotj8tOwzAURL1hgQofwKr-gZTrxI5jdhBelVIqoUoso9v4prVoEmRMafj6msJqpNHoaA5jVwJmslAKrtEf3H6WZrEABak5Zy9vuHt3_YbfD989D1viC-oGP_IF_tANv6Nx6C0vhz7QIfDKdS7ElR--Nls-j6XHJrg98VdCGzEX7KzF3Sdd_ueErR4fVuVzUi2f5uVtlWCuTSJQoDQFNYhGkEVtbQta6BytwSw-JSFlqilXa6PWjYQWjARrLUGbF0DZhE3_sCeh-sO7Dv1Y_4rVJ7HsCHI-SPo</recordid><startdate>20231008</startdate><enddate>20231008</enddate><creator>Chen, Howard</creator><creator>Pasunuru, Ramakanth</creator><creator>Weston, Jason</creator><creator>Celikyilmaz, Asli</creator><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20231008</creationdate><title>Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading</title><author>Chen, Howard ; Pasunuru, Ramakanth ; Weston, Jason ; Celikyilmaz, Asli</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a679-1a1a498ecaa91eda7ddf07176ad9a3855e14427e65b95bc40f0940ddde0f680e3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Computer Science - Computation and Language</topic><toplevel>online_resources</toplevel><creatorcontrib>Chen, Howard</creatorcontrib><creatorcontrib>Pasunuru, Ramakanth</creatorcontrib><creatorcontrib>Weston, Jason</creatorcontrib><creatorcontrib>Celikyilmaz, Asli</creatorcontrib><collection>arXiv Computer Science</collection><collection>arXiv.org</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Chen, Howard</au><au>Pasunuru, Ramakanth</au><au>Weston, Jason</au><au>Celikyilmaz, Asli</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading</atitle><date>2023-10-08</date><risdate>2023</risdate><abstract>Large language models (LLMs) have advanced in large strides due to the effectiveness of the self-attention mechanism that processes and compares all tokens at once. However, this mechanism comes with a fundamental issue -- the predetermined context window is bound to be limited. Despite attempts to extend the context window through methods like extrapolating the positional embedding, using recurrence, or selectively retrieving essential parts of the long sequence, long-text understanding continues to be a challenge. We propose an alternative approach which instead treats the LLM as an interactive agent, allowing it to decide how to read the text via iterative prompting. We introduce MemWalker, a method that first processes the long context into a tree of summary nodes. Upon receiving a query, the model navigates this tree in search of relevant information, and responds once it gathers sufficient information. On long-text question answering tasks our method outperforms baseline approaches that use long context windows, recurrence, and retrieval. We show that, beyond effective reading, MemWalker enhances explainability by highlighting the reasoning steps as it interactively reads the text; pinpointing the relevant text segments related to the query.</abstract><doi>10.48550/arxiv.2310.05029</doi><oa>free_for_read</oa></addata></record>
fulltext fulltext_linktorsrc
identifier DOI: 10.48550/arxiv.2310.05029
ispartof
issn
language eng
recordid cdi_arxiv_primary_2310_05029
source arXiv.org
subjects Computer Science - Computation and Language
title Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-24T04%3A21%3A54IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-arxiv_GOX&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Walking%20Down%20the%20Memory%20Maze:%20Beyond%20Context%20Limit%20through%20Interactive%20Reading&rft.au=Chen,%20Howard&rft.date=2023-10-08&rft_id=info:doi/10.48550/arxiv.2310.05029&rft_dat=%3Carxiv_GOX%3E2310_05029%3C/arxiv_GOX%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true