AI Testing Tools for LLM Products: What Traditional QA Suites Cannot Check

Your test suite passes green on every pull request. A week later a customer forwards the answer your product gave them, and it cites a returns policy your company never published. Nothing failed, because nothing in the suite could read the sentence the model actually wrote.

That gap is the whole subject of this post. The phrase ai testing tools points at two different markets, and only one of them can catch that answer. We name the split below, then stay with the half that tests what a language model actually produces.

We will cover why a normal assertion cannot judge model output, what these tools check instead, how to score an answer with no single right value, and how to run them in CI without a runaway bill. A QA suite falls short for a mechanical reason, so we start there.

What “AI Testing Tools” Means in Two Different Markets

Search for ai testing tools and the results describe one market while an LLM team needs the other. Market one is AI powered test automation for conventional software: self healing selectors, generated test cases, and visual diffing that flags a moved button. It is a real category with real vendors, and the wrong page for a team shipping model output.

Market two is the one this post serves. These are tools that evaluate the output of an LLM product: the answer it wrote, the citation it attached, the tool it called. The unit of work is a generated response, the failure modes are semantic, and the scoring looks nothing like a pass or fail on a fixed string.

The search data backs the split without needing invented volume. Expanding the plain term through autocomplete surfaces one LLM related string, while llm testing tools and ai testing tools for llm return evaluation intent throughout. Those are the strings the suggestion endpoint returns, with no traffic estimate implied.

Naming this split is itself the useful part. The pages that rank for the term rarely separate the two markets, so a reader who needs evaluation gets sent through automation content that never mentions model output. That is a small gap with a real cost.

If you landed here for test automation on a normal web app, the self healing and codegen tools are your category, and this is not that page. Everyone still reading is testing a model. The rest of this post is for you, starting with why your existing assertions cannot do the job.

Why Traditional QA Suites Cannot Assert on LLM Output

A conventional test compares one actual value to one expected value. An equality check passes when they match and fails when they do not, which is exactly right for a function that returns a total or a status code. Model output breaks that contract, because a question can have a wide space of acceptable answers rather than a single one.

Point equality, regex, and snapshot assertions all inherit the same flaw against that space. A snapshot pins one phrasing, so the next valid rewording fails the test even though the answer is correct. Loosen the check to stop the false alarms and it starts passing answers that are fluent and wrong. Neither setting is safe.

Non-determinism makes this worse, and it is not simply a temperature dial. Thinking Machines had Qwen3-235B-A22B-Instruct-2507 complete the prompt “Tell me about Richard Feynman” 1,000 times at temperature zero and got 80 unique outputs.

The completions were identical for the first 102 tokens, then split at token 103 on Feynman’s birthplace, with 992 continuing Queens, New York and 8 continuing New York City.

Source: Thinking Machines, Defeating Nondeterminism in LLM Inference (2025).

The stated cause is a lack of batch invariance. The result depends on how the server batched concurrent requests, not on anything in your test. Read that carefully, because it reframes a whole class of flaky failures. An intermittent difference here is a real property of the system, not a defect in your suite.

None of this retires your existing test runner. Schema shape, required fields, valid JSON, a latency ceiling, and the structure of a tool call are all still deterministic, and they still belong in pytest. Keep that layer. The scored layer sits on top of it, and the two answer different questions.

This is the mechanical reason a QA suite cannot check an LLM product. There is no single expected value to compare against, so the tool that assumes one has nothing to assert on. Once you accept that, the next step is choosing what to measure, since there is no assertion left to repair.

What AI Testing Tools Check That a Test Suite Cannot

The testable surface splits into a handful of check classes, each with its own input and scoring. Factual support asks whether the answer is grounded in the retrieved context, which is why retrieval quality feeds directly into it. If your reranker changes which passages arrive, our rerankers for RAG guide covers that upstream step.

Instruction following checks whether the output obeyed the rules in the prompt, like a word limit, a required disclaimer, or a demanded JSON shape. Safety and policy compliance checks whether the text crossed a line your product draws, from leaking a secret to giving advice you do not allow. Both are semantic judgments with no fixed target string.

Format and tone cover whether the answer reads the way your brand needs and parses the way your code needs. Tool call correctness checks whether the model called the right function with valid arguments. Consistency across repeated runs asks whether the same input holds steady, and cost and latency budgets keep an answer from being right but too slow.

Each of these needs an input a plain assertion cannot supply, and a scoring method a plain assertion cannot run. The table lays that out check by check, so you can see why the failure is structural rather than a matter of writing a stricter regex. It maps that surface for you to reason from.

Two rows deserve a flag. Run to run consistency cannot be judged from a single execution at all, since variance only shows up across repeats. Tool call correctness is the one row where a deterministic check still does most of the work, because argument shape and allowed values are things code can verify exactly.

In practice you do not run all six on every case. You pick the checks that match the failure modes your product actually has, then wire those into the suite. A support bot leans on factual support and policy. A structured extraction task leans on format and tool call correctness.

How to Score an Output When There Is No Expected Value

Three families of scoring answer the question, and most real suites blend them. Deterministic code checks read the output and compute a verdict directly, with no model involved. Statistical or similarity measures compare the answer to a reference and score the distance. Model graded judging asks a separate language model to rate the output against a rubric.

The judge approach draws the most doubt, so anchor it to evidence. The MT-Bench paper reported GPT-4 judges agreeing with humans more than 80% of the time, which is about the rate humans agree with each other. Our LLM evaluation frameworks guide covers the metric families in depth.

That agreement is not a blank check. The same paper names the ways a judge misfires: position bias toward the first answer shown, verbosity bias toward longer responses, self enhancement bias toward its own style, and limited reasoning on hard problems. A judge is a measuring instrument, and instruments need calibration before you trust the reading.

That calibration is the practical rule. Before a judge gates a build, check its scores against a set of human labels and confirm they track. The worry it answers is circular scoring: you are asking a model to grade a model, with no independent read on the grader. Calibration against human labels is what breaks that circle.

The honest answer sits between blind trust and blind rejection. Put a deterministic layer underneath the judge, so schema, required fields, and hard rules fail fast without a model. Reserve the judge for the semantic call that code cannot make, and keep it calibrated. That structure earns the judge its place in the gate.

Match the family to the check. Semantic similarity suits drift detection, where you care that today’s answer resembles yesterday’s. A rubric judge suits open questions like helpfulness or policy. Deterministic code suits anything with a crisp definition, and it is the cheapest of the three to run.

Building the Golden Dataset Your Tests Run Against

Everything above depends on one asset almost no competitor page describes: a golden dataset. It is a curated set of cases your suite runs on every change, with each case pinned to a known good outcome or an acceptance rubric. Without it, your scores have nothing stable to run against and cannot catch a regression.

A useful row carries more than an input and an answer. It holds the input, the retrieval or context set, an expected answer or a rubric that defines acceptable, a label for the failure mode it guards, and a note on where the case came from. That last field is provenance, and it earns its keep at review time.

The best cases come from your own logs rather than your imagination. Start with real production failures, because a case drawn from an incident is the only kind guaranteed to matter to a user. Add edge cases you can reason about next, then adversarial cases that probe for the failure modes you fear. The order tracks each case’s value.

Let discipline set the size of the set. Start with a small blocking set that runs on every pull request, then grow a larger set for the nightly schedule. Retire cases that every version passes, since a case that never fails has stopped telling you anything. A golden set gets pruned as often as it gets grown.

Practitioners already run this pattern. A common CI setup keeps a golden set of prompts in the pipeline and fails the build when semantic similarity or output structure drifts too far from the pinned reference.

The subtle drift is the harder kind to catch, which is the whole reason the pinned set has to live in CI rather than in someone’s memory.

Running AI Testing Tools in CI Without Blowing the Budget

The operational question is where teams stall, and the fix is a two tier design. Keep one small suite of deterministic and cheap checks that blocks every pull request, so feedback stays fast. Run the full golden dataset, judges included, on a schedule rather than on every push. The hot path stays fast while the slow path goes deep.

Open tools already support this shape. promptfoo documents CI/CD integration for GitHub Actions, GitLab, Jenkins, and CircleCI, along with caching guidance that includes a cache path environment variable and concurrency control to stay inside provider rate limits. Caching matters here, because re-scoring an unchanged case on every run is where the bill quietly grows.

DeepEval is Apache-2.0 and plugs into Pytest, which lets the scored layer live inside the test runner your team already knows. That placement is practical. Engineers write eval cases the way they write unit tests, and CI runs them through the same command, so adoption does not need a second pipeline standing beside the first.

Licences are worth getting right before you commit. promptfoo is MIT, DeepEval is Apache-2.0, Ragas is Apache-2.0 with its repository now maintained under vibrantlabsai and the older explodinggradients link redirecting there, and OpenAI Evals is MIT per its license file.

Star counts and dates move, so treat those as live checks rather than facts to quote. The licence and the maintenance home are what gate adoption.

One practical gotcha shows how young this tooling still is. promptfoo supports a global repeat count today, but a per case repeat override, so you can rerun only the consistency cases without paying for repeats on every other case, is still an open feature request. Plan your consistency checks around what your tool supports today, not what its roadmap promises.

The budget rule underneath all of this is simple. Every model graded check costs a call, so spend those calls where they change a decision. Cheap deterministic checks guard the pull request, expensive judged checks run nightly, and caching keeps you from paying twice for the same answer.

Catching Regressions After a Model Version Change

Here is the scenario every team hits and no ranking page covers. Nothing in your repository changed, yet behaviour changed anyway, because the provider updated the model under you. Catching that needs production signal as much as a test run, which our LLM observability guide maps in full.

The trap is calling normal variance a regression. Tie this back to the non-determinism section: one differing run proves nothing, since a single output can vary at temperature zero. A regression shows up as a shifted distribution across the golden set, a drop in the pass rate on a mode you were guarding, or a new cluster of low scores.

The table sorts the common changes by what breaks first, what to re-run, and what actually counts as a regression. Read across from the change you just made, and it tells you where to look before you start diffing outputs by hand. The columns stay consistent with the checks defined earlier.

One caveat keeps a scorer from faking a regression. A since closed DeepEval issue reported Contextual Precision over penalising overlapping chunks, which can read like a quality drop without being one. Before you flag a regression, confirm the scorer itself did not change its verdict on identical text.

Gating Releases with Custom Evals in Future AGI

The method in this post needs a scored gate inside a pipeline you already run.

Future AGI Evaluation fits there through custom evals you call from your own CI. The documentation is explicit that Future AGI never triggers or runs your pipeline, so the evals run wherever your code already builds. You keep control of the pipeline.

You author a custom eval in the eval builder, then pick the type that fits. An LLM-as-Judge eval applies your criteria in a single pass, an Agent Evaluator reasons over multiple turns with tools, and a Code eval runs deterministic logic in a sandbox and calls no model. Given the same input, the Code eval always returns the same output.

Each eval produces an output type you can gate on: a pass or fail, a percentage score, or a fixed choice from a set you define. A score at or above your threshold counts as a pass.

From your CI you call the evaluate method, read the result and its reason, and assert on it like any other test.

The documented use cases line up with this post.

You can gate pull requests on quality, so a regression blocks or flags the merge before it lands, and you can compare versions in CI by tagging evals to a version and reading the results in one place. The gate stays yours, scored by rules you wrote.

What to Add to Your Test Suite This Sprint

Come back to the green suite that passed while a customer got a made up policy. It passed because every check in it compared strings, and not one could read whether the answer was true. The fix adds a second layer that scores the answer on the qualities a string match cannot see, rather than a bigger regex.

Start small enough to finish this sprint. Take last month’s three worst production answers, turn each into a golden case with a rubric for what good looks like, and run those on every pull request. Put the full set on a nightly schedule. That one afternoon of work moves you from hoping to measuring.

Keep the principle in view as the suite grows. Deterministic checks sit underneath for schema, shape, and hard rules, and scored checks sit on top for everything semantic. When you are ready to run those scored checks as a release gate, Future AGI is built to score them against a threshold you set.