Formulation Comparison for Timeline Construction using LLMs
Constructing a timeline requires identifying the chronological order of events in an article. In prior timeline construction datasets, temporal orders are typically annotated by either event-to-time anchoring or event-to-event pairwise ordering, both of which suffer from missing temporal information...
Gespeichert in:
Hauptverfasser: | , , , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | Constructing a timeline requires identifying the chronological order of
events in an article. In prior timeline construction datasets, temporal orders
are typically annotated by either event-to-time anchoring or event-to-event
pairwise ordering, both of which suffer from missing temporal information. To
mitigate the issue, we develop a new evaluation dataset, TimeSET, consisting of
single-document timelines with document-level order annotation. TimeSET
features saliency-based event selection and partial ordering, which enable a
practical annotation workload. Aiming to build better automatic timeline
construction systems, we propose a novel evaluation framework to compare
multiple task formulations with TimeSET by prompting open LLMs, i.e., Llama 2
and Flan-T5. Considering that identifying temporal orders of events is a core
subtask in timeline construction, we further benchmark open LLMs on existing
event temporal ordering datasets to gain a robust understanding of their
capabilities. Our experiments show that (1) NLI formulation with Flan-T5
demonstrates a strong performance among others, while (2) timeline construction
and event temporal ordering are still challenging tasks for few-shot LLMs. Our
code and data are available at https://github.com/kimihiroh/timeset. |
---|---|
DOI: | 10.48550/arxiv.2403.00990 |