Why 90% Accuracy in Text-to-SQL is 100% Useless

The eternal promise of self-service analytics

Introduction

From manual ledger entries to conversational AI, the tools for data analysis have transformed dramatically over the last three decades. I have been working in the Analytics space for over 20 years. Back then, it was not called “analytics”, it was “Business Intelligence” or even “Decision Support Systems” in older times. The terms change, from data warehouses to Big Data, to lakehouses, and now with AI, the essence and the eternal promise of self-service Analytics remains the same: extracting truth from data to empower users without relying on someone from the data team.

The Promise of Conversational Interfaces

With the advent of Large Language Models (LLMs), one use case I find fascinating is developing conversational interfaces to chat with databases (Text-to-SQL). The potential here is immense, promising to democratize data access across organizations. However, for this specific use case, the solution has to be binary. It either works or it doesn’t. An accuracy of 80% or even 90% is, unfortunately, not enough.

The Complexity of the RAG Pipeline

I started my research on this topic over one year and a half ago and it quickly became clear that orchestrating a robust Text-to-SQL RAG (Retrieval-Augmented Generation) application is not trivial. You need multiple components in your pipeline, working in perfect harmony:

  • An intent classifier to detect the goal of the question.
  • A vector database to store additional context (like business definitions) that the language models need.
  • An embeddings model to vectorize this additional knowledge.
  • A retrieval mechanism for the stored data.
  • Access to the database.
  • The ability to generate SQL in the specific dialect of the database.
  • And the ability to evaluate the results.

This last part, evaluation, I believe is often omitted or treated as an afterthought, but it is perhaps the most crucial component for ensuring the reliability needed in an enterprise setting.

BigQuery: A Case Study in Native AI Integration

Managing this complex pipeline often requires integrating multiple platforms. I was recently impressed by how BigQuery has introduced the merger of Analytics and Generative AI natively in their platform. You have the ability to work with your SQL in the BigQuery IDE and use Gen AI immediately without going to another platform or product.

The Missing Piece: Rigorous Evaluation

Going back to eval frameworks. Platforms like BigQuery simplify the architecture, but they do not automatically solve the accuracy problem. If we accept that Text-to-SQL must be binary (correct or incorrect), we need evaluation strategies that reflect the messy reality of enterprise data, not the pristine environments of academic or demo datasets.

Metrics That Matter

  1. The Shift to Execution Accuracy (EX): Modern standard is Execution Accuracy (EX). This metric executes both the predicted SQL and the “Gold” (ground truth) SQL against the actual database and compares the returned result sets.
  2. Focused Evaluation: This allows for a more nuanced comparison, checking if the target columns and values are correct, while being more lenient on extraneous data or row ordering.
  3. The “Soft-F1” Metric: Provides partial credit by calculating the overlap between the predicted and gold results.
  4. LLM-as-a-Judge: If execution is impossible, an advanced LLM can be prompted to compare the semantic logic of the predicted SQL against the Gold SQL.

Spider 2.0: The Enterprise Reality Check

Currently, there are three remarkable evaluation frameworks: Spider 2.0, BIRD (BIg Bench for LaRge-scale Database Grounded Text-to-SQL) and SynSQL (based on synthetic data). Spider 2.0 (released in conjunction with ICLR 2025) introduces significant complexities that break LLMs in production:

  1. Massive Scale: Enterprise schemas are huge.
  2. Dialect Diversity: Real companies use various databases, not just SQLite.
  3. External Knowledge: Business logic resides in documentation or project codebases, not the schema.
  4. The Agentic Workflow: Models the workflow of a modern data engineer.

Conclusion: The Binary Bar for Enterprise Data

Achieving 90% accuracy might be academically interesting, but in the enterprise, it is industrially useless. The bar is binary: it works or it breaks trust. As platforms like BigQuery simplify the integration of AI and data, it is imperative that we simultaneously adopt sophisticated evaluation methodologies and rigorous benchmarks like Spider 2.0.