Your AI ran valid SQL. Did it answer the question?

An AI can write SQL that parses, executes, and returns a plausible number—and still answer the wrong question. A successful query is evidence that the database accepted the statement. It is not evidence that the metric, time window, joins, or explanation were right.

The published description of Data Berlin’s Building Reliable Data Agents: Our Evaluation Framework, with Fabian Marino and Felix Pitterling at Idealo, separates numeric correctness, SQL validity, and answer completeness. That distinction inspired this walkthrough. Our source evidence is the video description, not a transcript; the example below is our own, not a reproduction of Idealo’s framework or an endorsement by its speakers.

You do not need a large evaluation platform to start checking that distinction. Here is a small, fictional Postgres fixture with five questions and known answers. Bufflehead can give an AI tool read-only access to the prepared tables. The expected results and acceptance criteria remain yours to define.

Define the question before grading the answer

For this example, “September sales” means the recorded order amount for orders whose status is paid and whose placed_at timestamp falls in September 2026, in UTC. All amounts are USD, excluding tax and shipping; refunds are outside this fixture. This is an order-date measure, not cash collected during September. A missing amount is unknown, not zero.

The reporting window starts at September 1, inclusive, and ends at October 1, exclusive. If your business reports in a different time zone or by payment date, change that contract first. Do not grade a query against an unstated definition of “revenue.”

Prepare a disposable database

Run this setup in a fresh, disposable Postgres database with an ordinary SQL client and an administrator credential, not through Bufflehead. It writes only fictional data. Then connect Bufflehead using a database role with SELECT access to these tables and no write grants. Keep database setup and permission grants separate from the AI’s read-only workflow.

CREATE SCHEMA answer_demo;

CREATE TABLE answer_demo.customers (
  customer_id integer PRIMARY KEY,
  name text NOT NULL
);
CREATE TABLE answer_demo.orders (
  order_id integer PRIMARY KEY,
  customer_id integer NOT NULL
    REFERENCES answer_demo.customers(customer_id),
  placed_at timestamptz NOT NULL,
  status text NOT NULL CHECK (status IN ('paid', 'pending')),
  amount numeric(12,2) CHECK (amount >= 0)
);
CREATE TABLE answer_demo.order_items (
  item_id integer PRIMARY KEY,
  order_id integer NOT NULL REFERENCES answer_demo.orders(order_id),
  sku text NOT NULL
);

INSERT INTO answer_demo.customers VALUES
  (1, 'Ada'), (2, 'Ben'), (3, 'Cy'), (4, 'Dana');

INSERT INTO answer_demo.orders VALUES
  (1, 1, '2026-09-01 00:00:00+00', 'paid', 100),
  (2, 1, '2026-09-10 12:00:00+00', 'paid', 50),
  (3, 2, '2026-09-30 23:59:59.999999+00', 'paid', 150),
  (4, 2, '2026-09-12 12:00:00+00', 'pending', 900),
  (5, 3, '2026-09-14 12:00:00+00', 'paid', NULL),
  (6, 1, '2026-10-01 00:00:00+00', 'paid', 700),
  (7, 1, '2026-08-31 23:59:59.999999+00', 'paid', 800),
  (8, 3, '2026-09-15 12:00:00+00', 'paid', 0),
  (9, 1, '2026-09-20 12:00:00+00', 'paid', 100);

INSERT INTO answer_demo.order_items VALUES
  (11, 1, 'WIDGET'), (12, 1, 'WIDGET'),
  (21, 2, 'WIDGET'), (31, 3, 'OTHER'),
  (91, 9, 'WIDGET');

The deliberate traps are a pending order, timestamps on both boundaries, two equal order amounts, two matching items on one order, a missing amount, a genuine zero, and a customer with no orders. Item records identify product membership; they do not contain line prices or quantities.

1. Totals: is USD 400 the whole answer?

Ask: “What were September paid-order sales, and is the amount complete?” This reference query returns the known subtotal alongside the number of qualifying orders and missing amounts.

SELECT COUNT(*) AS paid_orders,
       COUNT(amount) AS known_amounts,
       COUNT(*) FILTER (WHERE amount IS NULL) AS missing_amounts,
       SUM(amount) AS known_sales
FROM answer_demo.orders
WHERE status = 'paid'
  AND placed_at >= TIMESTAMPTZ '2026-09-01 00:00:00+00'
  AND placed_at < TIMESTAMPTZ '2026-10-01 00:00:00+00';

Expected: 6 paid orders, 5 known amounts, 1 missing amount, and USD 400.00 in known sales. “September sales were $400” is incomplete. A passing explanation says the known amounts total $400, but one qualifying order has no amount, so the complete total cannot be established from these records.

Postgres aggregates ignore null inputs for SUM and COUNT(amount); COUNT(*) counts rows. With no qualifying rows, SUM returns null. The query must carry the missing-data evidence into the explanation.

2. Rankings: first among whom?

Ask: “Rank customers with September paid orders whose amounts are all known; list any customer withheld for missing amounts.” Do not quietly turn an incomplete customer subtotal into a definitive rank.

WITH totals AS (
  SELECT customer_id,
         SUM(amount) AS known_sales,
         COUNT(*) FILTER (WHERE amount IS NULL) AS missing_amounts
  FROM answer_demo.orders
  WHERE status = 'paid'
    AND placed_at >= TIMESTAMPTZ '2026-09-01 00:00:00+00'
    AND placed_at < TIMESTAMPTZ '2026-10-01 00:00:00+00'
  GROUP BY customer_id
), ranked AS (
  SELECT customer_id,
         DENSE_RANK() OVER (ORDER BY known_sales DESC) AS sales_rank
  FROM totals
  WHERE missing_amounts = 0
)
SELECT c.name, t.known_sales, t.missing_amounts, r.sales_rank
FROM totals t
JOIN answer_demo.customers c USING (customer_id)
LEFT JOIN ranked r USING (customer_id)
ORDER BY r.sales_rank NULLS LAST, c.customer_id;

Expected: Ada is rank 1 at $250; Ben is rank 2 at $150; Cy has a known subtotal of $0 but one missing amount and no rank. Dana has no qualifying orders and is outside this ranking. Ada leads the complete-data subset—not necessarily every customer. Equal complete totals would share a rank; the customer ID only stabilizes display order.

3. Date boundaries: which orders actually belong?

Ask: “Show the order IDs included in the September paid-order window.” A count alone can hide the wrong membership.

SELECT order_id
FROM answer_demo.orders
WHERE status = 'paid'
  AND placed_at >= TIMESTAMPTZ '2026-09-01 00:00:00+00'
  AND placed_at < TIMESTAMPTZ '2026-10-01 00:00:00+00'
ORDER BY order_id;

Expected: exactly 1, 2, 3, 5, 8, and 9. Order 1 is included at the opening instant; order 3 is included at the end of September; order 6 is excluded at October 1; order 7 is too early; order 4 is pending. An inclusive upper boundary would admit order 6. An end-of-month cutoff rounded to whole seconds could drop order 3.

4. Joins: did one order become two?

Ask: “What is the total order amount for September paid orders containing WIDGET?” This asks for whole-order amounts, not revenue attributable to WIDGET line items. Those are different metrics.

SELECT COUNT(*) AS matching_orders,
       SUM(o.amount) AS known_order_amount,
       COUNT(*) FILTER (WHERE o.amount IS NULL) AS missing_amounts
FROM answer_demo.orders o
WHERE o.status = 'paid'
  AND o.placed_at >= TIMESTAMPTZ '2026-09-01 00:00:00+00'
  AND o.placed_at < TIMESTAMPTZ '2026-10-01 00:00:00+00'
  AND EXISTS (
    SELECT 1 FROM answer_demo.order_items i
    WHERE i.order_id = o.order_id AND i.sku = 'WIDGET'
  );

Expected: 3 orders, $250, and no missing amounts. Order 1 has two WIDGET items, but its $100 amount belongs in the total only once. A direct matching-items join followed by SUM(o.amount) returns $350. “Fixing” that with SUM(DISTINCT o.amount) returns $150, because orders 1 and 9 legitimately have the same amount.

Use order identity, not distinct money values, to avoid duplicate counting. EXISTS checks membership without multiplying the outer rows. If you need item-level revenue instead, obtain the line amounts and their business definition; this fixture cannot answer that question.

5. Missing data: zero, unknown, or no orders?

Ask: “Show every customer, including those with no September paid orders, and distinguish incomplete amounts.” Keep the order filters in the join condition so customers without a match remain visible.

SELECT c.name,
       COUNT(o.order_id) AS paid_orders,
       COUNT(o.amount) AS known_amounts,
       COUNT(o.order_id) FILTER (WHERE o.amount IS NULL)
         AS missing_amounts,
       SUM(o.amount) AS known_sales,
       CASE
         WHEN COUNT(o.order_id) = 0 THEN 'no_orders'
         WHEN COUNT(o.order_id) > COUNT(o.amount)
           THEN 'missing_amount'
         ELSE 'complete'
       END AS data_status
FROM answer_demo.customers c
LEFT JOIN answer_demo.orders o
  ON o.customer_id = c.customer_id
 AND o.status = 'paid'
 AND o.placed_at >= TIMESTAMPTZ '2026-09-01 00:00:00+00'
 AND o.placed_at < TIMESTAMPTZ '2026-10-01 00:00:00+00'
GROUP BY c.customer_id, c.name
ORDER BY c.customer_id;

Expected: Ada has 3 orders, $250, complete; Ben has 1 order, $150, complete; Cy has 2 orders, one known zero and one missing amount, so the $0 known subtotal is incomplete. Dana has 0 orders, a null sum, and status no_orders. Reporting “Cy and Dana spent zero” loses two important distinctions.

For the unmatched customer, COUNT(o.order_id) is zero; COUNT(*) would count the placeholder row produced by the left join. Moving the paid-order conditions into WHERE would remove that customer altogether. See Postgres’s table-expression documentation for how joins and filtering interact.

Turn the fixture into an answer check

Give your AI tool the metric definition and one question at a time through its configured Bufflehead connection. Ask it to show the generated SQL, supporting rows or counts, and a plain-English conclusion. Keep the expected answers separate until you assess the response.

• SQL validity: did the statement execute against the intended schema within the permitted read scope?
• Result correctness: did values and row membership match the reference? Compare results, not exact SQL text; different queries can be equivalent.
• Answer completeness: did the response state USD, the UTC September window, paid status, missing amounts, and any restricted comparison population relevant to the question?

Record failures separately. An execution error, a double-counted total, and a correct subtotal described as complete need different fixes. Save the prompt, tool/model version, generated SQL, results, and explanation when comparing runs. Recheck after changing prompts, schemas, or model versions; five passing questions do not establish a general accuracy rate.

For this walkthrough, we executed the five reference queries against the fixture in isolated PostgreSQL 14, in read-only transactions, and checked their expected results. That validates these examples—not the performance of an AI model or an end-to-end Bufflehead agent run.

Where Bufflehead fits

Bufflehead is the read-only bridge to existing structured data. It does not provide a built-in evaluation suite, define your business metrics, fill in missing values, or guarantee that an AI’s explanation is accurate. The useful combination is scoped access, explicit definitions, inspectable SQL, and answers checked against evidence.

Start with a handful of questions whose answers you already know. If an agent cannot distinguish $400 in known amounts from a complete $400 total, giving it more tables is unlikely to solve the underlying problem.

Download Bufflehead to connect to existing data. For the access side of the workflow, read Read-only SQL is not a data-access policy; for choosing useful context, read Stop pasting your database into the prompt.