Text-to-SQL Calibration: No Need to Ask -- Just Rescale Model Probabilities
Calibration is crucial as large language models (LLMs) are increasingly deployed to convert natural language queries into SQL for commercial databases. In this work, we investigate calibration techniques for assigning confidence to generated SQL queries. We show that a straightforward baseline -- de...
Gespeichert in:
Hauptverfasser: | , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | Calibration is crucial as large language models (LLMs) are increasingly
deployed to convert natural language queries into SQL for commercial databases.
In this work, we investigate calibration techniques for assigning confidence to
generated SQL queries. We show that a straightforward baseline -- deriving
confidence from the model's full-sequence probability -- outperforms recent
methods that rely on follow-up prompts for self-checking and confidence
verbalization. Our comprehensive evaluation, conducted across two widely-used
Text-to-SQL benchmarks and multiple LLM architectures, provides valuable
insights into the effectiveness of various calibration strategies. |
---|---|
DOI: | 10.48550/arxiv.2411.16742 |