Most AI failures in organizations are not dramatic. An assistant cites a policy that was replaced last year. A summary drops the one exception that mattered. A drafting tool repeats sensitive text it should not have seen. Each of these is preventable with testing that happens before the tool is relied on, not after a complaint.

Start with the real work

Generic benchmarks say little about how a tool will behave on your tasks. Build a test set from the work the tool will actually do: real questions staff ask, real documents they summarize, real forms they fill in. A few dozen well-chosen cases, each with an agreed correct answer or a clear standard, are worth more than thousands of synthetic ones.

Test five things

  • Accuracy. Does the answer match the agreed correct answer, and how often?
  • Factuality and sources. Is every claim supported by a source the user can open? Answers without sources should be treated as unverified.
  • Compliance. Does the output follow your rules on tone, required language, records and privacy?
  • Prompt injection. Can text hidden in a document or web page make the tool ignore its instructions?
  • Data leakage. Can a user get the tool to reveal information they are not allowed to see?

Agree the bar before you test

Decide in advance what result is good enough, and who signs off. A tool that drafts internal notes can tolerate more error than one that answers the public. Writing the threshold down first stops a team from talking itself into a launch after seeing the results.

Keep a person where the risk is

For tasks with real consequences, design the workflow so a person reviews the output before it is used. This is not a sign that the AI failed; it is how responsible teams use it while confidence builds.

Keep testing after launch

Models, documents and rules change. Re-run the test set when any of them changes, sample live answers for review, and give staff an easy way to flag a wrong answer. Testing is not a gate you pass once; it is part of running the tool.

← All insights