Measuring AI answer accuracy against your own records

Accuracy needs a reference

Reading what an assistant says about a company and judging it plausible is not measurement. A measurement compares each claim to a reference that was established independently of the assistant. That reference is a record of facts, each with a source, the person who confirmed it and the date.

Where the record is incomplete, the finding is not accuracy. It is a list of questions the company cannot yet answer itself, which is a more useful result than it sounds.

Choose the questions a buyer asks

The question set determines what the measurement can see. For a manufacturer, distributor or dealer, three groups cover most of the exposure.

• Identity questions: who is authorized to sell this line, where is the nearest location, what departments does it have.
• Product questions: what is the part number for this application, what are the dimensions and ratings, what supersedes this item.
• Operational questions: is it available, how soon, can it be quoted, who do I contact.

Probe identifiers and measurements

Ask for values that can only be right or wrong: a part number, a model year range, a nominal and actual dimension, a unit. Vague questions produce vague answers that are hard to score. Specific probes also expose a common failure: an assistant states a plausible value in the right format that does not exist.

Include probes that should return unknown. A question about a product the company does not make, or a fact the record does not hold, tests whether the assistant says so or invents an answer.

Score claims, not answers

An answer contains several claims. Split it and judge each one against the record: correct, incorrect, unsupported, or correctly stated as unknown. A single answer can be mostly right and still carry the one wrong claim that matters, such as the wrong part number inside a correct description.

Record the assistant, the date, the exact question and the full answer, so a later run can be compared with this one and a change can be read as a trend instead of a surprise.

Repeat on a schedule

Answers change as models, sources and retrieval change, and so does the record. A single run is a snapshot. The useful artifact is the series: which claims were wrong in each run, which were corrected, and which regressed. A claim that returns after being fixed points to an upstream feed that keeps reintroducing it.

Each wrong claim then goes through tracing it to its source. Lehvel's AI Observatory runs this method across ChatGPT, Claude, Gemini and Perplexity against a client's checked record.

Questions