DataVinci: Learning Syntactic and Semantic String Repairs
String data is common in real-world datasets: 67.6% of values in a sample of 1.8 million real Excel spreadsheets from the web were represented as text. Systems that successfully clean such string data can have a significant impact on real users. While prior work has explored errors in string data, p...
Gespeichert in:
Hauptverfasser: | , , , , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | String data is common in real-world datasets: 67.6% of values in a sample of
1.8 million real Excel spreadsheets from the web were represented as text.
Systems that successfully clean such string data can have a significant impact
on real users. While prior work has explored errors in string data, proposed
approaches have often been limited to error detection or require that the user
provide annotations, examples, or constraints to fix the errors. Furthermore,
these systems have focused independently on syntactic errors or semantic errors
in strings, but ignore that strings often contain both syntactic and semantic
substrings. We introduce DataVinci, a fully unsupervised string data error
detection and repair system. DataVinci learns regular-expression-based patterns
that cover a majority of values in a column and reports values that do not
satisfy such patterns as data errors. DataVinci can automatically derive edits
to the data error based on the majority patterns and constraints learned over
other columns without the need for further user interaction. To handle strings
with both syntactic and semantic substrings, DataVinci uses an LLM to abstract
(and re-concretize) portions of strings that are semantic prior to learning
majority patterns and deriving edits. Because not all data can result in
majority patterns, DataVinci leverages execution information from an existing
program (which reads the target data) to identify and correct data repairs that
would not otherwise be identified. DataVinci outperforms 7 baselines on both
error detection and repair when evaluated on 4 existing and new benchmarks. |
---|---|
DOI: | 10.48550/arxiv.2308.10922 |