LLM Regression Testing: Catch Silent Model-Swap Failures

Last week the feature worked. This week the same prompts come back worse, the tests you have still pass, and nothing in your code changed. No error fired. The only thing that moved was the model version sitting behind a floating alias.

That is a silent model-swap failure, and it is exactly what LLM regression testing exists to catch. A model you did not choose to change can quietly lower the quality of a shipped feature, with no exception and no deploy to point at.

This guide is a build-it walkthrough. We cover what LLM regression testing is, why a model swap fails without a signal, and a runnable pytest harness plus a CI gate you can copy. Everything here targets the failure a version change causes.

One scope note up front. The examples pin a model id and assert against a versioned golden set, so the setup catches a swap before it reaches users. Your test runner and CI stay yours, and the harness just gives them something worth checking.

TL;DR

• LLM regression testing runs a fixed, versioned golden set through your system and asserts the outputs still clear a known-good bar, so a model, prompt, or code change cannot quietly lower quality.
• Outputs are non-deterministic, so assert properties and eval scores against a threshold rather than exact strings, and assert on the golden-set average rather than any single generation.
• The failure it exists to catch is the silent model swap: a floating alias resolves to a new version, quality drops, and nothing raises an exception.
• Pin dated snapshots in production, and treat a forced deprecation migration as a change that has to pass the suite.
• Run the suite on every pull request, on every model or provider version change, and on a schedule. The scheduled run is the only trigger that catches a provider-side swap.
• GPT-4 fell from 84.0% to 51.1% on the same prime-versus-composite set between March and June 2023. Only a check on outputs would have caught it.

What LLM Regression Testing Is

LLM regression testing runs a fixed set of inputs through your LLM system and asserts the outputs still clear a known-good bar. The point is that any change, to the model, the prompt, or the surrounding code, cannot quietly lower quality without a test going red.

The obvious objection comes first. LLM outputs are non-deterministic, so how do you test them at all. You do not assert exact strings for open-ended answers. You assert properties and eval scores against a threshold, and those hold steady even when the wording shifts run to run.

That distinction is the whole trick. A regression test on a generative system checks a stable signal rather than a literal output. For a classifier or an extraction task you can still match strings, and for free text you check that a score stays above the bar you set.

So the suite is a fixed battery of cases with an assertion attached to each one. You run it whenever something could have changed, and a failure tells you quality dropped on a specific input. That is the safety net a shipped LLM feature needs, and the rest of this guide builds it piece by piece.

The Silent Model-Swap Failure

A model swap fails silently when the model behind your feature changes and nothing errors, so quality drops with no signal to alert on. This is the precise failure LLM regression testing exists to catch, and it is the one most suites miss because they check code, not the model underneath.

There is a well-known, dated example. A study by Chen (Stanford), Zaharia (UC Berkeley), and Zou (Stanford), later published in Harvard Data Science Review, ran a 1,000-question prime-versus-composite set against two GPT-4 snapshots and found accuracy fall from 84.0% in March 2023 to 51.1% in June 2023. The same model name returned materially different outputs three months apart.

That number survived a public correction, which is why it is worth quoting. The first version of the paper tested 500 primes and nothing else, and reported a 97.6% to 2.4% collapse. Narayanan and Kapoor pointed out that an all-prime set cannot separate a real accuracy drop from a flipped answer bias, because a model that simply answers “composite” every time scores near zero on it. The authors rebuilt the set with 500 composites added, and the balanced result is the 84.0% to 51.1% above. The confusion matrix in the final paper shows the March version getting primes and composites both mostly right, while the June version called an integer composite 99.7% of the time. On a balanced two-way question, 51.1% is a coin flip.

The paper names a cause, and it is the one that should worry you. Chain-of-thought prompting lifted GPT-4 from 59.6% to 84.0% in March. In June the same prompt bought 0.1 points, because the new version stopped following the instruction to think step by step: its average answer length fell from 638.3 characters to 3.9. Nothing in the caller’s code changed. A prompting technique that had been carrying the feature simply stopped working.

The correction made the lesson sharper, not softer. A stable model name went from usable to chance on the same questions with no error raised, and only a check on outputs would have flagged it. Related silent decay shows up in reward model drift, where a scorer stops tracking quality with no weight change.

The triggers are worth naming, because each one changes behavior without raising anything. The table lists them. Not one raises an exception, and a golden-set eval assertion is what catches all four.

Every trigger routes through the same weakness. Your system trusts a model id to mean a fixed behavior, and that assumption quietly stops being true. The defense is to stop trusting the id and start checking the output on a set of cases you control.

Why Model Aliases Break Silently

An alias is a bare model name that floats to whatever version the provider currently serves. A pinned snapshot is a dated id that stays fixed until the provider deprecates it. That difference decides how a change reaches you, quietly or with a deadline attached.

With an alias you get silent upgrades. The provider ships a new version, your alias resolves to it, and your feature starts behaving differently the next time it runs. Nothing in your code or your logs marks the moment the behavior changed, which is what makes it so hard to trace.

With a pinned snapshot you trade that for a deprecation deadline. The behavior holds steady while the snapshot is live, and when the provider retires it you get a dated notice instead of a surprise. You choose when to migrate, and you can run the suite before you flip the switch.

The practical rule follows from that. Pin snapshots in production, track deprecation notices as real work, and treat any forced migration as a change that must pass the regression suite before it ships. A migration you did not choose is still a change, and it deserves the same gate as your own edits.

Build the Regression Harness

The harness has three parts. A versioned golden dataset that pairs each input with the assertion it must satisfy, a function that calls your pinned model, and a parametrized test that runs every case. Keep the dataset in version control, so a change to it shows up in review like any other diff.

Parametrize is the part that pays off. Each golden case becomes its own reported test with its own id, so a failure names the exact input that regressed instead of collapsing everything into one red mark. When a swap breaks three cases, you see precisely which three.

# test_regression.py  ->  run with: pytest -q
# A versioned golden set drives one test per case, so a failure names the
# exact input that regressed after a model or prompt change.
import json
import pytest

with open("golden.json") as f:
    GOLDEN = json.load(f)          # [{"input": ..., "must_contain": ...}, ...]

def model_answer(prompt: str) -> str:
    # Replace with your client. PIN the model id, never a floating alias.
    raise NotImplementedError("Wire your pinned model client here")

@pytest.mark.parametrize("case", GOLDEN, ids=lambda c: c["input"][:40])
def test_no_regression(case):
    out = model_answer(case["input"])
    assert case["must_contain"] in out, (
        f"Regression on {case['input']!r}: "
        f"missing {case['must_contain']!r}")

The must_contain assertion in that harness is the simplest one that works. It fits deterministic and keyword outputs, where a correct answer has to include a specific string. For open-ended text it is too blunt, and the scoring section further down replaces it with an eval score against a threshold.

The model_answer stub is where you wire your client, and the comment there carries the load-bearing rule. Pin the model id, never a floating alias. A regression suite that calls an alias is testing a moving target, and it will pass one day and fail the next for reasons that have nothing to do with your code.

Keep the golden file small enough to run fast and pointed at cases that matter. A handful of high-value inputs you check on every change beats a giant set you run once a quarter. The next sections cover what belongs in that file and how to score outputs that have no single right answer.

Gating a Model Swap in CI

A harness only protects you if a regression cannot merge past it. Run the suite on every pull request, and let a failing golden case block the merge the same way a failing unit test would. The workflow below does exactly that on GitHub Actions.

# .github/workflows/llm-regression.yml
name: LLM regression
on:
  pull_request:
  schedule:
    - cron: "0 6 * * *"   # catches a provider-side swap with no deploy
jobs:
  regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt pytest
      - run: pytest -q          # a failing golden case blocks the merge
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

The mechanics are ordinary CI. Check out the repo, set up Python, install pytest, and run the suite, with the provider key supplied as a secret. A non-zero exit from pytest fails the job, and a failed required check stops the merge. Nothing here is LLM-specific except what the tests assert.

The assertion you pick depends on the output type, so the suite usually mixes several layers. The table lays them out from strictest to most tolerant, so you can match each golden case to the check that fits its output.

Pick the loosest layer that still catches a real regression. A JSON endpoint wants a schema check, a keyword answer wants a substring match, and open-ended prose wants a score threshold with a human spot check on a sample. Stacking the right layers per case is what makes the gate trustworthy rather than noisy.

Choosing Golden Cases That Catch Regressions

A golden set is a curated list of the inputs you cannot afford to get wrong, not a random sample of traffic. Three kinds of case earn a place. The highest-traffic inputs, the ones that have broken before, and the edge cases that expose format or safety sensitivity.

Think of it as a liability list. Every case is there because a regression on it would cost you something real, a support ticket, a broken integration, a safety issue. That framing keeps the set focused on consequence instead of coverage for its own sake. Our guide to golden-set design goes deeper on selection and labeling.

The maintenance rule is what keeps it alive. Every production incident becomes a new golden case, so the suite grows toward the failures you actually see rather than the ones you imagined at the start. A bug that reached users once should never reach them the same way twice.

Size is a tradeoff you set on purpose. A tight set of high-value cases runs on every pull request without slowing anyone down, and a larger set can run on a schedule. What matters is that every case in the fast set is there because its failure would genuinely hurt.

Scoring Beyond Exact Match

For open-ended outputs, swap the substring assertion for an evaluator that returns a score, then assert that score stays above a threshold. This is the move that makes regression testing work for generative text, where there is no single correct string to match against.

Two evaluator types cover most needs. An LLM-as-a-judge rubric grades qualities like helpfulness, groundedness, or instruction adherence on a scale. Deterministic checks handle format, required fields, and forbidden content, where the answer is a clean pass or fail. Many suites run both and combine them.

Assert on the aggregate, not the single case. Compute the score across the whole golden set and require that the average does not fall below your baseline, so one noisy generation does not fail the build while a real broad drop still does. Pair that with deterministic metrics for the checks that must never regress.

Set the threshold from history, not a guess. Run the evaluator on known-good outputs, see where the score lands, and set the bar a little below that with room for run-to-run noise. Too tight and the gate cries wolf, too loose and a real regression slips through untouched.

How Often Should You Run LLM Regression Tests?

Run LLM regression testing on three triggers, and each one catches a different kind of change. On every pull request, so a code or prompt edit cannot merge with a regression. On every model or provider version change, so a swap is gated before it ships to anyone.

The third trigger is the one teams forget. Run the suite on a schedule against your production models, even when nothing on your side changed. A silent model-swap happens without a deploy, so a scheduled run is the only way to notice a provider-side change before your users do.

Match the cadence to the risk. A daily scheduled run is enough for most features, and a high-stakes path can run hourly. The point is that a check tied only to your own deploys will miss every change that originates upstream, which is exactly the failure this whole guide is about.

So the schedule has two halves. Change-triggered runs guard what you ship, and time-triggered runs guard what the provider ships. Together they close the gap that a deploy-only check leaves wide open.

Which LLM Regression Testing Tool Should You Use?

The harness above is deliberately dependency-free, because the assertion pattern matters more than the tool. Once the suite grows, the choice comes down to where you want the assertion to live.

The pytest-native tools keep everything in your repository, which is the shortest path when your golden set is small and your checks are string-shaped. The hosted tools earn their place once you want run history, side-by-side comparison, and judge-based scores you did not have to write yourself.

Then read the last column, because a model swap is not an ordinary regression. Every row here will tell you that a golden case got worse. The question worth asking of any of them is what happens after CI goes green, since a golden set proves the candidate handled your cases, not that it handled your traffic. That gap is why the Future AGI section below ends on shadow experiments rather than on another way to write assertions.

Building the Golden Set From Production Traces

Writing golden cases by hand is where most suites stall. The better source is already sitting in your traces, because the cases that matter are the ones that already failed.

The mechanism to look for is grouping rather than logging. Error Feed reads a sample of the traces in an Observe project, works out what went wrong in each one, and groups the traces that failed the same way into a single issue you work like a ticket.

That grouping is what makes curation tractable. One issue standing for forty similar failures gives you one golden case to add, not forty near-duplicates that slow the suite and test the same thing. Add the case, keep the trace id in a comment, and the next model swap has to clear a bar set by a real incident.

LLM Regression Testing with Future AGI

The method this guide recommends is to assert an eval score in CI against a versioned golden set, and to compare a candidate model against a baseline before you promote it. Future AGI’s custom evals and its datasets and experiments are built to be exactly those two layers.

With custom evals you define the grading rule, an LLM judge or a deterministic check, select the dataset columns to grade, and set a pass or fail threshold. Then you call it from your own CI, so a score below the bar blocks the merge.

The evaluation docs walk through defining a rule, and built-in evaluators such as groundedness and instruction adherence are there when you want a ready scorer.

Datasets and experiments cover the baseline comparison. Run the same golden set against a candidate model and your current baseline, then read the scores side by side before you promote a swap.

An experiment runs every prompt-and-model combination you configure against the same dataset rows and scores them with the same evals, so the columns are directly comparable. Add the candidate model as a second configuration and pick the winner before you promote a swap. The CI/CD guide covers running those evals on every pull request, and the ai-evaluation SDK with the open-source traceAI instrumentation lets you run it all in your own stack.

The strongest check happens after CI, because the swap that breaks you is the one you never tested under real load. Shadow experiments silently mirror live requests to a candidate model without affecting users, so you compare cost, latency, and quality on real traffic before you switch. A golden set tells you the candidate handled your cases. This tells you it handled your users. For a model swap, that is the check worth having, and it is the one a repo-only suite cannot give you.

The scope is deliberate. Future AGI is the eval-and-compare layer, and your test runner and CI stay exactly where they are. It scores and it compares, so pytest and GitHub Actions keep doing what they already do well.

What to Test Before Your Next Model Swap

Come back to the feature that degraded for no visible reason. An alias floated to a new version, the behavior shifted, and no error marked the moment. Only a golden-set assertion, run on a schedule, would have caught it before your users did.

The setup that prevents a repeat is short to state. Pin snapshots so upgrades stop being silent. Keep a versioned golden set of the inputs you cannot afford to break. Assert eval scores in CI on every pull request and every version change. Run a scheduled check against production models, and turn each incident into a new case.

That list is the whole defense. It costs a small suite and a few minutes of CI, and it converts a silent model-swap from an outage your users report into a red check you catch in a pull request.

When you want the eval gate and the baseline comparison off your own plate, Future AGI’s custom evals and experiments give you both, and the CI/CD guide has the workflow.