I built a prototype for a natural language query agent a few months ago. It basically revolves around you typing a business question in plain English and getting an answer right out of the data platform. I was thinking that the main challenge of such project would be in generating the SQL.
I was wrong.
The hard part is everything that must happen before SQL.
The AI can write SQL just fine. The actual nightmare is deciding what the system is allowed to do before it even writes a single line of code.
Most enterprise AI today is basically just a copilot. It helps you write an email or summarize notes. If it messes up, you just ignore it. Agents are different. They don’t wait around for a human to hit approve. They pull the data, run queries, and trigger workflows all on their own.
If a normal BI dashboard has a mistake on it, the team just gets confused. But if an autonomous agent acts on that same mistake, it might trigger a fake alert, update the wrong database record, or push a terrible business decision downstream before anyone even notices.
The First Step Is To Understand Business, Not SQL
When a user asks something like “What was our monthly revenue?”, it is very tempting to pass that question straight into an LLM and ask it to generate SQL.
But that is exactly where many systems go wrong.
SQL should not be the first step. Understanding the question should be the first step.
The first step must be intent detection and metric mapping. In a company, be it big or small, the term “revenue” is a total mess. Does it mean recognized revenue from finance? Net revenue after refunds? Or booked revenue from the orders table? If the agent just guesses behind the scenes, the SQL might run perfectly and return a number, but the actual business meaning behind it will be completely wrong.
If the system has different definitions for revenue, the agent should not guess. It needs to stop and explicitly ask the user which definition they mean. That little pause is not a failure. It is just good governance.
Another point to remember is you also absolutely cannot point an agent at production tables. These tables would be full of old fields and confusing internal names.
The orders table may contain various date fields, including the order creation date, the placement date, the payment date, the fulfillment date, and the cancellation date.
Which one should the agent use to calculate monthly revenue? A human analyst knows because they would have the context.
An agent needs that rule explicitly written down in a semantic layer. That layer consists of a strict contract, telling the agent exactly what to do. It will also say to use the recognized revenue table, group it by calendar month, exclude internal test accounts, and subtract refunds. Without those rules, the agent is just forced to guess based on column names.
Anusha Kovi, the author of this text, is going to be a speaker during the upcoming AI Loves Data San Francisco event. Secure your spot today!
A simple metric contract could look like this:
metric: monthly_revenue
owner: finance_analytics
source_table: finance.recognized_revenue
time_column: revenue_recognition_date
grain: month
filters:
- payment_status = 'completed'
- is_test_account = false
- is_cancelled = false
adjustments:
- subtract_refunds = true
allowed_dimensions:
- region
- product_line
restricted_columns:
- customer_email
- payment_methodNo Querying Until Access and Quality Are Clear
Another problem is about permissions. This is where many deployments go wrong.
In a normal BI product, access follows the person using the tool. Once I open a dashboard, the system knows which data is visible to me and which isn’t. That permission check is not separate from the experience. It is part of how the product works.
But with agents, teams often take a shortcut. They connect the agent to a broad service account because the agent needs enough access to answer many different types of questions. This is extremely dangerous. Wide service access should never become wide user access. Even if the agent is connected through a powerful backend service account, the actual query still needs to run with the permissions of the person asking the question.
So, if a finance analyst asks, “What was our monthly revenue?”, the system should only use the data that the finance analyst is allowed to see.
The agent should not get extra visibility just because it is an agent. This matters because sensitive data can leak before the final answer is even produced. The leak can happen while the agent is doing ‘innocent’ things, like pulling sample rows, inspecting schemas, checking column values, or building context for the prompt.
We also need to rethink how we handle data quality. Right now, a lot of data quality work is reactive. A pipeline breaks, an alert fires, and then an engineer goes in to investigate. That may be acceptable for a dashboard. It is not acceptable for an agent. With an agent, the bad data may already have been used to decide, trigger a workflow, or send a recommendation. By the time the alert fires, the damage may already be done.
Let’s say an agent is supposed to summarize daily revenue at 9 AM.
On a normal day, the revenue table finishes loading by 8 AM. But today, maybe because of heavy load on the cluster, the pipeline is delayed, and the table only contains partial data from yesterday. A human looking at a dashboard might see a freshness warning and know to wait.
An agent needs that same warning built into its own process. Before it runs the query, it should check whether the table is fresh and whether the last pipeline completed successfully. If the data is not ready in the tables, the agent should stop. It should not produce a confident answer from bad data.
That check can be simple at first:
def check_table_health(table_status):
if not table_status.last_pipeline_success:
return "BLOCK", "Last pipeline run failed"
if table_status.is_stale:
return "BLOCK", "Table is not fresh"
if table_status.has_open_quality_incident:
return "BLOCK", "Open data quality incident"
return "ALLOW", "Table is safe to use"Auditability Is Not Optional
At some point, an AI agent will make a strange decision. It may return the wrong number, choose the wrong metric, or answer a question that should have gone to a human.
A regular server log will be inadequate in such a case. While the log may indicate that the problem happened at 2:05 PM, it will not explain why it happened. To debug an agent effectively, a full trace is required. You need to view the user’s request, how the agent understood it, which metric it chose, what rules it followed, the SQL it created, the data retrieved, and how the concluding answer was formulated. In the absence of this trace, you aren’t truly debugging. You are guessing. And when the compliance team asks what happened, ‘the model just did it’ is not an acceptable answer.
You also need to treat different actions based on risk. Reading a metric or drafting a report may be low or medium risk. But updating a database, changing a customer record, or approving a refund is high risk. Those actions should require human approval.
If the agent is allowed to make changes in the database, then it is imperative that the rollback should be made ready in the system. The old value, new value, and the mechanism by which the old value can be restored are to be recorded for any sort of problem. Some people think that governance affects the growth of AI. In real production systems, it does the opposite. Without governance, agents remain stuck in demo mode. Teams may like the prototype, but they will not trust it with customer records or production workflows.
Governance is what makes trust possible. Autonomy should not be given to an agent on day one. It should be earned step by step through traceability, approval, monitoring, and rollback.
Author: Anusha Kovi.
Anusha Kovi is going to be a speaker during the upcoming AI Loves Data San Francisco event. Secure your spot today!