What Is ASR Accuracy?

What Is ASR Accuracy?

ASR accuracy is a voice AI evaluation metric that measures how correctly an automatic speech recognition system turns spoken audio into text. In production it shows up in the eval pipeline, LiveKit or Pipecat traces, and downstream agent logs when a wrong transcript causes wrong reasoning. Teams usually measure it with word error rate, named-entity misses, and confidence by cohort. FutureAGI uses ASRAccuracy to compare transcripts against ground-truth audio or reference text before voice agents ship.

Why It Matters in Production Voice Agents

Bad ASR accuracy makes every downstream component look worse than it is. A user says “cancel order A-1042”, the transcript becomes “candle order eight 1042”, and the LLM either calls the wrong tool or asks an irrelevant clarification. In healthcare, finance, logistics, and support, the most expensive errors are often small: a name, dosage, card number, airport code, or negation.

The pain splits across roles. Developers debug prompts even though the root cause is upstream transcription. SREs see higher retry rates, longer calls, and p99 time-to-first-audio drift because the agent burns turns repairing misunderstood input. Product teams see lower task completion on noisy mobile calls. Compliance teams lose auditability when the stored transcript is not faithful to the audio.

The production symptoms are measurable: rising word error rate, low transcription confidence, more “can you repeat that?” prompts, tool arguments with impossible IDs, and support escalations clustered by accent, channel, or background noise. For 2026-era voice agents, ASR accuracy is not just an audio metric. It is the first reliability gate in a multi-step pipeline where one bad transcript can poison retrieval, tool selection, policy checks, and the final spoken answer.

How FutureAGI Measures ASR Accuracy

FutureAGI maps the anchor eval:ASRAccuracy to the ASRAccuracy evaluator, a speech-to-text accuracy eval in the fi.evals surface. In a voice-agent workflow, engineers run it in three places: offline on a golden audio dataset, during LiveKitEngine simulations before release, and on sampled production traces from runtimes instrumented with traceAI integrations such as livekit or pipecat.

A practical setup looks like this. A team keeps reference audio and expected transcripts for checkout, password reset, and appointment-booking calls. Each nightly simulation produces an audio file, raw ASR transcript, normalized transcript, final agent response, and call-level outcome. ASRAccuracy scores the raw transcript against the reference. The dashboard slices the score by locale, noise condition, microphone channel, and ASR vendor. If WER crosses 6% for a cohort or named-entity misses double week over week, CI blocks the ASR model rollout or opens a regression task.

FutureAGI’s approach is to connect the ASR score to the agent trace, not leave it as an isolated speech benchmark. The same trace can show ASRAccuracy, TaskCompletion, ToolSelectionAccuracy, and stage latency side by side. Unlike a standalone Whisper or Deepgram benchmark, this lets an engineer answer the production question: did transcript error actually change the agent’s decision? The next action is concrete: alert, pin the previous ASR provider, add accent-specific test cases, or route noisy calls to a safer fallback flow.

How to Measure or Detect ASR Accuracy

Measure ASR accuracy as a layered signal, not a single scoreboard number:

• ASRAccuracy: FutureAGI evaluator that scores speech-to-text accuracy against ground-truth audio or reference text.
• Word error rate (WER): substitutions, deletions, and insertions divided by reference words; track by cohort, not only globally.
• Named-entity error rate: misses on names, SKUs, amounts, addresses, medications, dates, and account identifiers.
• Transcription confidence distribution: useful for triage, but calibrate it against labeled calls before alerting.
• Trace correlation: join ASR score to TaskCompletion, ToolSelectionAccuracy, escalation rate, and p99 time-to-first-audio.
• User feedback proxy: repeat-request rate, “wrong transcript” tags from QA, and human takeover after misunderstood intent.

from fi.evals import ASRAccuracy

asr = ASRAccuracy()
result = asr.evaluate(
    audio_path="calls/order-change.wav",
    ground_truth="I need to change order A-1042."
)
print(result.score)

A practical alert is “ASRAccuracy below threshold for two consecutive cohorts,” not “one bad call failed.” Pair threshold alerts with sampled call review so teams can separate accent drift, microphone quality, and ASR model regressions.

Common Mistakes

The common errors are not academic. They are usually measurement shortcuts that hide production damage:

• Treating aggregate WER as production truth. It hides accent, language, channel, and background-noise cohorts that break real calls.
• Optimizing ASR accuracy without named-entity checks. Low WER can still miss dollar amounts, medications, ticket IDs, and names.
• Accepting vendor confidence as a quality score. Confidence is model-calibrated uncertainty, not end-to-end transcript correctness against ground truth.
• Scoring clean studio audio only. Phone codecs, barge-in, packet loss, and overlapping speakers change the error profile.
• Evaluating ASR after the LLM rewrites text. Measure raw ASR output before normalization, summarization, tool extraction, or retrieval.