What Is Voice Agent Regression Detection?

What Is Voice Agent Regression Detection?

Voice agent regression detection is the process of catching new quality drops in a voice AI agent by comparing current calls with an approved baseline. It is a voice reliability workflow that appears in simulation suites, CI eval gates, and production call traces. FutureAGI teams store baseline scenarios in fi.datasets.Dataset, replay them through LiveKitEngine, and compare ASRAccuracy, AudioQualityEvaluator, TaskCompletion, latency, and escalation metrics before a model, prompt, ASR, or TTS change reaches callers.

Why It Matters in Production LLM and Agent Systems

Voice agents regress in ways that text evals rarely catch. A new ASR model may turn “close my savings account” into “close my savings discount.” A prompt change may make the agent over-confirm every step. A TTS provider update may add 700ms to time-to-first-audio. Each looks small in isolation, but the caller experiences a slower, less accurate, less trustworthy system.

The named failure modes are transcript drift, turn-taking regression, TTS degradation, and false task completion. Developers feel it as flaky voice-agent builds that pass chat tests. SREs see p99 time-to-first-audio, jitter, silence duration, and retry count move after a provider or routing change. Product teams see lower completion for one locale, device class, or noisy-channel cohort. Compliance teams lose evidence when a regulated call is marked resolved but the agent skipped a required escalation.

This is especially relevant for 2026-era agentic voice stacks because a spoken request often triggers multiple downstream actions: ASR, turn detection, retrieval, tool selection, policy checks, model response, and TTS. One upstream regression can produce a correct-looking transcript with the wrong tool call. Logs usually show repeated corrections, longer handle time, low transcription confidence, barge-in spikes, or rising human-transfer rate. Regression detection turns those symptoms into a release gate instead of a support queue surprise.

How FutureAGI Detects Voice Agent Regressions

FutureAGI’s approach is to keep voice baselines as evaluable datasets, not screenshots of passing calls. An engineer creates a fi.datasets.Dataset for approved scenarios with columns such as scenario_id, cohort, expected_transcript, expected_outcome, baseline score, and baseline latency. The same scenarios are replayed through the simulate-sdk LiveKitEngine, which captures audio and transcripts for each simulated call.

The workflow then attaches evaluators with Dataset.add_evaluation. ASRAccuracy checks the speech-to-text boundary. AudioQualityEvaluator catches clipping, silence, and noise issues before the LLM sees text. TTSAccuracy can be used for generated speech checks when spoken output must match a planned response. TaskCompletion verifies the call goal, while ToolSelectionAccuracy helps when the agent picks tools such as account lookup, booking change, or escalation.

A real regression gate might compare the candidate build against the last approved release across 1,000 support calls. The gate fails if ASRAccuracy drops by more than two points for noisy mobile callers, if p99 time-to-first-audio exceeds 900ms, or if TaskCompletion falls on refund calls. The engineer opens the failing cohort, replays the audio, checks ASR and agent spans, and rolls back the ASR provider, fixes the prompt, or adjusts the turn-detection threshold.

Unlike transcript-only QA in a Vapi review queue, FutureAGI keeps the baseline dataset, replayed audio, transcript, evaluator score, and outcome trace together.

How to Measure or Detect Voice Agent Regression Detection

Measure regressions as deltas from an approved baseline, sliced by scenario and cohort:

• ASRAccuracy: speech-to-text score; alert on drops by accent, locale, microphone, carrier, or noise condition.
• AudioQualityEvaluator: audio-quality score for clipping, silence, and noise before transcript scoring.
• TTSAccuracy: generated speech check for cases where the spoken reply must match the intended text.
• TaskCompletion: call-level outcome score; catches “polite but unresolved” calls.
• Dashboard signals: p99 time-to-first-audio, barge-in rate, turn-detection error rate, eval-fail-rate-by-cohort, and tool retry count.
• User proxies: transfer-to-human rate, repeated correction rate, thumbs-down rate, reopened tickets, and post-call escalation.

Minimal fi.evals shape:

from fi.evals import ASRAccuracy

asr = ASRAccuracy()
score = asr.evaluate(
    input="cancel my second card",
    output="cancel my secured card",
).score
assert score >= 0.97

The important practice is comparison. A score of 0.94 may be acceptable for a noisy public-transit cohort and unacceptable for a clean banking authentication call.

Common Mistakes

Most teams miss voice regressions because they compare the wrong artifact or average away the failure:

• Testing replayed text instead of replayed audio. Text replay misses ASR, endpointing, jitter, background noise, and TTS queueing.
• Comparing only global averages. Overall WER can stay flat while a Spanish-accent, mobile, or headset cohort fails.
• Using stale baselines after product changes. A new policy or call flow needs a new approved baseline, not suppressed alerts.
• Ignoring latency regressions. A correct answer after a long silence behaves like failure in a live call.
• Skipping tool and escalation checks. Good ASR does not prove the agent selected the right tool or escalated regulated requests.