First Heuristic Then Rational: Dynamic Use of Heuristics in Language Model Reasoning
Multi-step reasoning instruction, such as chain-of-thought prompting, is widely adopted to explore better language models (LMs) performance. We report on the systematic strategy that LMs employ in such a multi-step reasoning process. Our controlled experiments reveal that LMs rely more heavily on he...
Gespeichert in:
Hauptverfasser: | , , , , , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | Multi-step reasoning instruction, such as chain-of-thought prompting, is
widely adopted to explore better language models (LMs) performance. We report
on the systematic strategy that LMs employ in such a multi-step reasoning
process. Our controlled experiments reveal that LMs rely more heavily on
heuristics, such as lexical overlap, in the earlier stages of reasoning, where
more reasoning steps remain to reach a goal. Conversely, their reliance on
heuristics decreases as LMs progress closer to the final answer through
multiple reasoning steps. This suggests that LMs can backtrack only a limited
number of future steps and dynamically combine heuristic strategies with
rationale ones in tasks involving multi-step reasoning. |
---|---|
DOI: | 10.48550/arxiv.2406.16078 |