AI for data work: text-to-SQL, analysis, and the columns that don't exist

"Ask your data a question in plain English" is one of the most compelling AI demos, and one of the easiest to get quietly, dangerously wrong. The model turns your question into SQL that runs, returns a number, and is confidently incorrect. It guessed your schema. Here's how to get data answers people can actually act on.

The failure that looks like success

The model doesn't know your database. So when you ask "how many customers churned last quarter," it hallucinates: a churned column that doesn't exist, a join on the wrong key, the wrong date field, the wrong grain. The output is syntactically perfect SQL that executes and returns a number.

The worst data bug runs clean and returns a plausible wrong number. Nobody catches an error that never happens; the wrong number just rides straight into a board deck.

Making text-to-SQL reliable

• Give it the schema. This is the whole game. Table and column names, types, relationships, and descriptions. Most text-to-SQL failures come down to one thing: it guessed the schema and guessed wrong. For a big database, retrieve the relevant tables into context instead of dumping all 800.
• Ground it in real metadata (actual names, sample values, documented joins, and your semantic layer / metric definitions if you have one). "Active user" means something specific to your business; the model can't infer it.
• Verify before you trust it. Show the SQL, run it on a sample, sanity-check the number against something you already know. A data answer you can't trace back to a query you've read is a guess wearing a result's clothes.
• Read-only, scoped, bounded access. The model writes queries; you run them through a read-only, row-limited, timeout-bound connection. Never give an LLM write access to a production database. Treat its generated SQL as untrusted input, because that's what it is.

Beyond SQL: code-interpreter analysis

For analysis past a single query, the reliable pattern is the model writing Python (pandas), running it, reading the result, and iterating: a code-execution loop. It beats freehand reasoning about data because the code is the work, and you can inspect exactly what it did. Same discipline applies: review the code, not just the conclusion.

The trap that isn't the model's fault

The subtlest failures are semantic, not syntactic. The SQL runs fine; the business logic is wrong. Which timestamp defines "last quarter"? Does "customer" include trials? These are knowledge problems, not modeling ones. The fix is a semantic layer or documented metric definitions the model can ground in, not a smarter model. This is the knowledge-vs-behavior distinction: the model doesn't need training. It needs your definitions in context.

The honest take

AI is excellent for exploration and first-draft queries, getting you to a working query and a hypothesis fast, and dangerous as an unverified source of numbers people decide on. Use it to draft, explore, and unblock, ground it in your real schema and metrics, run it read-only, and verify before anyone acts. And if you're shipping a text-to-SQL feature, build an eval set of question→correct-SQL pairs and run it on every change. Because "it worked in the demo" and "it's right on the questions that matter" are very different claims.