Making SQL Executable with Execution-Guided Text-to-SQL: A Reproducible Study on WikiSQL-Style Queries

Abstract

Natural-language interfaces to relational databases require models that generate SQL which is not only syntactically valid, but also executable and semantically faithful to the user question. In practice, text-to-SQL systems often fail for reasons that are invisible to purely string-based evaluation: a query may differ from the gold SQL yet still execute to the correct answer, or it may match the gold form while being brittle to schema and value variation. This paper investigates how simple execution feedback can be used to make generated SQL executable and to improve answer accuracy. We present a controlled comparison of three decoding settings that mirror common LLM prompting pipelines: (i) prompt-only decoding with minimal schema awareness, (ii) schema-linking decoding that explicitly maps question spans to table columns and cell values, and (iii) execution-guided decoding that executes candidate SQL programs and repairs empty/invalid results using value-level feedback. Experiments follow the WikiSQL task formulation introduced by Seq2SQL [1] (single-table SELECT with optional aggregation and up to three WHERE conditions). We evaluate on the standard split sizes (56,355/8,421/15,878) and report both exact-match (EM) and execution accuracy (EX). On the test split of our WikiSQL-style reproduction, prompt-only decoding reaches 30.02% EX, schema-linking reaches 82.84% EX, and execution-guided decoding reaches 84.13% EX, a +1.28 absolute improvement over schema-linking. Detailed breakdowns show that execution feedback primarily reduces condition-value errors and improves robustness as query complexity increases. The study highlights when execution-based filtering/repair is beneficial, and provides a reproducible baseline for evaluating “make-it-executable” decoding strategies.