SParC: Cross-Domain Semantic Parsing in Context

Paper · Source
LLM ArchitectureDomain Specialization in LLMs

We present SParC, a dataset for cross-domain Semantic Parsing in Context. It consists of 4,298 coherent question sequences (12k+ individual questions annotated with SQL queries), obtained from controlled user interactions with 200 complex databases over 138 domains. We provide an in-depth analysis of SParC and show that it introduces new challenges compared to existing datasets. SParC (1) demonstrates complex contextual dependencies, (2) has greater semantic diversity, and (3) requires generalization to new domains due to its cross-domain nature and the unseen databases at test time. We experiment with two state-of-the-art text-to-SQL models adapted to the context-dependent, crossdomain setup. The best model obtains an exact set match accuracy of 20.2% over all questions and less than 10% over all interaction sequences, indicating that the crossdomain setting and the contextual phenomena of the dataset present significant challenges for future research. The dataset, baselines, and leaderboard are released at https:// yale-lily.github.io/sparc.

Introduction. Querying a relational database is often challenging and a natural language interface has long been regarded by many as the most powerful database interface (Popescu et al., 2003; Bertomeu et al., 2006; Li and Jagadish, 2014). The problem of mapping a natural language utterance into executable SQL queries (text-to-SQL) has attracted increasing attention from the semantic parsing community by virtue of a continuous effort of dataset creation (Zelle and Mooney, 1996; Iyyer et al., 2017; Zhong et al., 2017; Finegan-Dollak et al., 2018; Yu et al., 2018a) and the modeling innovation that follows it (Xu et al., 2017; Wang et al., 2018; Yu et al., 2018b; Shi et al., 2018). While most of these work focus on precisely mapping stand-alone utterances to SQL queries, generating SQL queries in a context-dependent scenario (Miller et al., 1996; Zettlemoyer and Collins, 2009; Suhr et al., 2018) has been studied less often.

Discussion / Conclusion. In this paper, we introduced SParC, a largescale dataset of context-dependent questions over a number of databases in different domains annotated with the corresponding SQL representation. The dataset features wide semantic coverage and a diverse set of contextual dependencies between questions. It also introduces unique challenge in mapping context-dependent questions to SQL queries in unseen domains. We experimented with two competitive context-dependent semantic parsing approaches on SParC. The model accuracy is far from satisfactory and stratifying the performance by question position shows that both models degenerate in later turns of interaction, suggesting the importance of better context modeling. The dataset, baseline implementations and leaderboard are publicly available at https:// yale-lily.github.io/sparc.

Synthesis notes from this paper's topics 8

This paper has no direct semantic matches yet — here are notes from the topic(s) it's filed under.

Lines of inquiry this paper opens 24

Research framings built by reading the notes related to this paper — the questions it feeds into.

What distinguishes genuine understanding from correct linguistic performance? What makes reasoning models struggle with ill-posed and unfamiliar problems? Do language models truly understand their encoded knowledge or merely retrieve patterns? What determines whether visible reasoning traces improve task performance? Does learning articulatory physics explain speech communication superiority? How do frontier models develop behaviors that escape detection? How can parameter sparsity improve interpretability while maintaining model capability? What determines how neural networks organize their internal knowledge representations? How should test-time compute be allocated adaptively across task difficulties? Does prompting inject new knowledge into models or activate existing knowledge? Do language models learn underlying linguistic structure or merely surface patterns? What explains the gap between LLM semantic understanding and production? How do pretraining data composition and task diversity shape reasoning? What are the trade-offs between memory capacity and continuous task learning? What reasoning limitations and biases do language models systematically exhibit? Do AI explanations represent actual model reasoning or constructed post-hoc narratives? How does explicit structure improve LLM reasoning on complex tasks? Can prompts override training-embedded biases in language models?