Blog 2026-10-10

Can you trace that AI answer back to the SQL?

An AI answer is easier to review when its question, SQL and returned rows stay together. If the final number is wrong, that record helps you determine whether the query selected the wrong population or the summary misrepresented the result.

DATA-UNI’s “Slack + BigQuery + Langfuse: Build and Monitor an AI Data Agent (Gemini + MCP)”, uploaded October 8, prompted this example. We reviewed its description, not a loaded transcript, and did not reproduce its workflow. Slack integration, Langfuse tracing and deployment controls belong to that setup; no endorsement or Bufflehead tracing integration is implied.

A question with a known answer

Use four fictional orders: two paid orders on October 8 worth 5,000 and 3,000 USD cents, one pending order that day worth 2,000 cents, and one paid order the next day worth 9,000 cents. Ask: “What were paid order totals for October 8, 2026, in USD cents?” The known answer is 8,000 cents ($80). This measures stored order totals, not accounting revenue or settled cash.

An illustrative parameterized BigQuery query is:

SELECT SUM(total_cents) AS total_cents
FROM `demo.shop.orders`
WHERE status = @status AND order_date = @day

The intended parameters are status = paid (STRING) and day = 2026-10-08 (DATE). This SQL is proposed for a separately prepared BigQuery fixture; it has not been executed against BigQuery. Parameter binding depends on your execution path. Bufflehead’s current query method accepts SQL text; this example does not claim that it exposes the parameterized call shown above.

Keep a review record

Save the question, exact executed SQL, typed parameters or literal values, start time, elapsed time, columns and units, returned rows, completeness status and the actual final response. Identify the backend and fixture version. For a BigQuery job, capture the job identifier and available execution metadata through the external runner. Record failures and truncation explicitly; a partial page is not the full result.

question: paid order totals for October 8, in USD cents
parameters: status=paid; day=2026-10-08
rows: [{total_cents: 8000}]
result_complete: true
summary: $80 in paid order totals

This compact record is illustrative. Timestamps and elapsed times in our downloadable records belong to the local test, not a cloud query. An execution record supports investigation; it is not automatically a tamper-proof audit log or proof that the business definition was correct.

Two different failures

First, remove the status predicate. The query now includes the pending order and returns 10,000 cents ($100). A summary saying $100 accurately repeats the rows, but answers the wrong question. Inspecting the SQL reveals the missing paid-order condition.

Second, keep the correct query and its 8,000-cent result, but pair it with an authored summary claiming $100. Here the query matches the fixture’s definition; the summary contradicts its rows. Preserve the returned data rather than relying on a model’s account of what the tool returned.

A reviewer can classify these failures only because the question, query, result and summary are available together. If the rows are absent, either failure can look like the same unexplained wrong number.

What we tested

We executed equivalent SQL against a disposable in-memory SQLite fixture using Python’s standard library, with query-only mode enabled. Checks passed for the correct 8,000-cent total, the incorrect-population 10,000-cent total, detecting a deliberately inconsistent summary, excluding the next day, and refusing a write. The script and three sample records make the local demonstration reproducible.

This is an offline test of review-record contents, not a BigQuery execution, Bufflehead integration test or live AI evaluation. Both incorrect summaries are authored examples. We have not measured model accuracy, cloud latency or tracing overhead.

Use the record on a real run

Prepare synthetic data outside Bufflehead, then ask your AI client the known-answer question through its read-only database connection. Save the SQL actually submitted and the tool results actually returned. Record any pagination or limits, and compare the final response with the expected $80. Keep the model and fixture fixed when comparing runs; a useful trace does not itself demonstrate an accuracy improvement.

Bufflehead supplies read-only access to existing BigQuery data. The review record, model tracing and evaluation checks in this example belong to external instrumentation. Keep records scoped to what a reviewer needs: real query text and rows may contain private data, so choose retention and access controls appropriate to that data.

Try Bufflehead with a known-answer question. When an answer looks suspicious, follow the record from the question to the SQL, from the SQL to the rows, and from the rows to the summary.

Try it on your own data.

Native binary, no Docker, no cloud egress. Free for local files and direct database connections — request a demo for the AWS SSM path.

Download free Request a demo