Using AI as an analyst, and checking its work

AI tools such as ChatGPT, Gemini and Claude can write SQL and Python, suggest charts, clean text columns and draft the summary of a report. Used well, they make an analyst faster. Used blindly, they produce neat-looking answers that are wrong, and the person who sends the report is the one who gets blamed, not the AI.

What you will do: run a real analysis of a sales file with an AI assistant, then check its numbers independently with a short Python script. You will leave with a repeatable checking routine.

Prerequisites: basic SQL or pandas, and the idea of metrics from *Business metrics and KPIs*. You need a free AI tool (ChatGPT, Gemini or Claude all have free plans, with limits) and Python with pandas (Google Colab has both). You will use the sales.csv file generated in *Project: sales dashboard for a coffee chain*; a small file with the same columns also works.

A key rule: let the AI write code, and let the code compute the numbers. A language model predicts text. When it computes a total itself, it is guessing what a total would look like. When it writes a query that you run, the database does the arithmetic exactly. Some AI tools can run code themselves in a sandbox; even then, ask to see the code so you can check the logic.

AI cannot see your table. If you only say "analyse my sales", it guesses column names and invents a structure. Give it the schema, one example row, the business question and what each column means. Tell it the traps, such as one order spanning several rows.

You are helping me, a data analyst, analyse sales data. I will run all code myself and send you results.

Table: sales (one row per item in an order)
Columns: order_id (int, an order can have several rows), order_date (date), store (text), item_id (text), item_name (text), category (text: coffee, tea, bakery, food), unit_price (int, rupees), qty (int), revenue (int, = unit_price * qty)
Example row: 1001, 2026-01-03, Indiranagar, cappuccino, Cappuccino, coffee, 180, 2, 360

Question: [for example: which store has the lowest average order value, and why might that be?]

First, tell me how you will define each metric and any assumptions. Then write the pandas code (assume the file is sales.csv). Do not state any numbers until I give you the output.

What to expect: A good answer defines average order value as total revenue divided by distinct orders (not the mean of the revenue column), uses the exact column names, and does not invent figures.

Checking means getting the same answer a different way or testing it against something you know. The script below is a basic sanity pass you can run on any sales table: it checks row counts, missing values, that revenue equals price times quantity, and recomputes headline totals by two different methods.

import pandas as pd

df = pd.read_csv("sales.csv", parse_dates=["order_date"])

print("Rows:", len(df))
print("Total revenue:", df["revenue"].sum())
print("Distinct orders:", df["order_id"].nunique())
print("Average order value:", round(df["revenue"].sum() / df["order_id"].nunique(), 2))

print()
print(df.groupby("store")["revenue"].sum().sort_values(ascending=False))

monthly = df.groupby(df["order_date"].dt.to_period("M"))["revenue"].sum()
print()
print(monthly)

# data quality checks
print()
print("Rows where revenue != unit_price * qty:", (df["revenue"] != df["unit_price"] * df["qty"]).sum())
print("Missing values:", df.isna().sum().sum())

# a second way to get total revenue: add up the per-store totals
by_store_total = df.groupby("store")["revenue"].sum().sum()
print("Total from store totals matches overall total:", by_store_total == df["revenue"].sum())

Save it next to sales.csv and run python check.py. Requires pandas (pip install pandas).

Now compare these numbers with what the AI's code or text claimed. If AI wrote a number that its own code does not produce, trust the code. If code and your check disagree, find out why before sharing anything. The usual causes are below.

AI is also a useful reviewer. Show it your finished query and result, and ask it to find holes. It may raise false alarms, so judge each point yourself, but it often spots a missing filter or an unclear definition.

Act as a sceptical senior analyst. Review my analysis below. Do not rewrite it; list problems.

Question: [question]
Metric definitions: [definitions]
Code: [paste code]
Result: [paste output]
My conclusion: [your conclusion]

Check for: double counting, wrong denominators, missing filters, unclear time periods, outliers, claims of cause the data cannot support, and anything that would make this result different on a longer time range. Rank the issues by how much they could change the conclusion.

What to expect: Look for concrete, specific issues. Ignore vague advice. Test each real issue with code before changing your conclusion.

Explain this SQL to a beginner, line by line, and then tell me what could go wrong with it (for example duplicate rows after a join or NULL handling):

[paste query]

What to expect: Useful for learning and for checking that the AI's query really does what the question needs.

Companies now ask how you use AI at work. A strong answer is: "I use it to draft queries and explain errors, I always run the code myself and check the totals with an independent method, and I do not share confidential data with unapproved tools." Some interviews give a task with AI allowed; they are judging whether you verify, not whether you can copy.