What Is a Regression Eval?

What Is a Regression Eval?

A regression eval is a fixed evaluation suite — a stable golden dataset plus a stable set of evaluators — that you run against every candidate release. Its only job is to surface whether the new version performs worse than the old version on the cases you’ve already decided matter. The dataset must not change between runs (or the comparison is invalid), and the evaluators must be deterministic enough that score deltas signal real quality changes, not evaluation noise. It is plain regression testing applied to LLM outputs.

Why Regression Evals Matter in Production

LLM systems regress more often than backend systems do, because more knobs change more often. A prompt edit, a model version bump, a retriever index rebuild, a tool-schema change, a temperature tweak — any of these can flip behavior on cases that previously worked. Without a regression eval, the team only finds out when a user complains, and by then the bad version has been in production for hours or days.

Pain across roles: ML engineers spend Friday afternoons bisecting which of three merged PRs caused the quality drop. Product managers can’t trust release notes — “we improved tone” might also mean “we broke citations.” SREs see latency or cost shift but lack the quality dimension to know if the trade is worth it. Compliance can’t show consistent behavior across audits because nothing was captured at each release point.

For 2026 agent stacks, regression evals are doubly important because changes compound. Updating the planner prompt may improve goal completion but tank tool-selection accuracy; only a multi-evaluator regression suite catches the trade-off. The headline number (“agent did better!”) hides the regression on a sub-metric. FutureAGI’s approach: never collapse to a single number for regression eval. Always show per-evaluator deltas plus the aggregate. Comparable workflows in LangSmith ship regression suites but require manual setup of the dataset versioning and CI gates; we ship those primitives as SDK calls.

How FutureAGI Handles Regression Evals

FutureAGI’s approach is to make regression evals a single SDK invocation in CI. fi.datasets.Dataset lets you pin a specific dataset version. Dataset.add_evaluation() attaches the suite. CI runs the evaluation, gets back a per-row, per-evaluator score table, and compares against the previous green run. AggregatedMetric produces the headline pass/fail; per-evaluator deltas are surfaced in the run report.

For ongoing regression hygiene, every traceAI integration (traceAI-openai-agents, traceAI-langchain, traceAI-llamaindex) auto-samples production into a configurable dataset. Promoting a sampled trace to the golden set is a one-line operation. Failures from the human annotation queue (fi.queues.AnnotationQueue) also flow back into the golden dataset so the regression suite reflects real production failure modes, not just synthetic ones.

A real flow: a team running an agent on traceAI-openai-agents maintains a 1,500-row golden dataset (mix of synthetic edge cases and promoted production failures). On every PR, GitHub Actions calls the SDK to run six evaluators — TaskCompletion, Groundedness, ToolSelectionAccuracy, StepEfficiency, IsHelpful, and a CustomEvaluation for brand voice — with thresholds on each plus a 0.85 aggregate gate. A model upgrade from a smaller variant to a larger one improved aggregate by 0.04 but dropped StepEfficiency by 0.12 (more tool calls per task). The diff caught it pre-merge; the team kept the old model on cost-sensitive routes and shipped the new one on a small route via the Agent Command Center’s weighted-routing.

How to Measure or Detect Regression Eval Health

A regression eval is itself a system that needs monitoring:

• Per-evaluator delta vs. previous green run: track every metric, not just the aggregate.
• Eval suite runtime: a regression suite that takes 90 minutes will get bypassed; budget ≤15 minutes for CI relevance.
• Flaky-eval rate: % of evaluators that swing >0.05 on a re-run with the same inputs. Above 5% means tighten judge temperature or switch to a sturdier evaluator.
• Coverage: % of recent production failure modes represented in the golden dataset. <80% means the dataset is stale.
• Dataset version freshness: when was the last new row added? Promote weekly.

Minimal Python:

from fi.datasets import Dataset
from fi.evals import Groundedness, TaskCompletion, AggregatedMetric

golden = Dataset.get("agent-golden", version="v23")
results = golden.add_evaluation(
    AggregatedMetric([Groundedness(), TaskCompletion()], weights=[0.5, 0.5]),
    threshold=0.85,
)
assert results.passed, results.report

Common Mistakes

• Mutating the golden dataset. Adding rows is fine; deleting or editing existing rows breaks the comparison. Version every change.
• Single aggregate score only. A regression in one sub-metric can be hidden by improvements in another. Always track per-evaluator deltas.
• Running regression eval on the dev model only. Run against the exact model + prompt + retriever build that will deploy.
• Ignoring evaluator noise. Judge-based evaluators have variance; rerun N=3 and take the mean for high-stakes gates.
• No alerting on cohort regressions. Aggregate may pass while one user cohort silently regresses 15%.