What Is Prosody?

What Is Prosody?

Prosody in voice AI is the pattern of pitch, stress, rhythm, volume, pauses, and speaking rate that shapes how a synthesized or agent voice sounds to a listener. It is a voice-AI quality signal, not a transcript metric, and it shows up in TTS regression tests, LiveKit simulations, and production call traces after text becomes audio. FutureAGI treats prosody as a conceptual voice-quality dimension today, measured through nearby audio, tone, timing, and human-rating signals rather than a dedicated Prosody evaluator.

Why Prosody Matters in Production LLM and Agent Systems

Flat or misplaced prosody turns a correct answer into a bad call. A voice agent can choose the right tool, follow the policy, and read the right text, yet sound rushed, bored, sarcastic, or alarmed at the wrong moment. The named failure modes are emotion mismatch and turn-timing failure. Emotion mismatch makes a billing apology sound indifferent or a fraud warning sound casual. Turn-timing failure makes the agent pause too long, cut off the caller, or place emphasis on the wrong word.

The pain shows up across teams. Developers see “works in transcript” bugs because text review misses pitch contour, stress, and pause placement. SREs see higher barge-in rate, longer call duration, TTS provider regressions, and p99 time-to-first-audio spikes after voice changes. Product teams see repeated prompts such as “are you still there” or “can you repeat that.” Compliance teams worry when consent language, refund terms, or medical instructions are technically present in the transcript but delivered too fast to be understood.

In 2026 voice-agent systems, prosody is tied to multi-step pipelines: ASR, intent routing, retrieval, tool calls, policy checks, LLM response generation, TTS, interruption handling, and post-call summaries. A prosody failure can be downstream of the prompt, the TTS provider, the selected voice, a locale setting, a latency workaround, or bad endpointing. Logs usually show it indirectly: rising escalation rate, negative call ratings, turn overlap, repeat requests, and voice-quality drops isolated to one provider, accent, language, or scenario.

How FutureAGI Handles Prosody

Because the current FAGI anchor for prosody is none, FutureAGI’s inventory does not list a dedicated Prosody evaluator or an eval:Prosody surface. FutureAGI’s approach is to treat prosody as a composed voice-quality signal that must be evaluated with the audio artifact, transcript, scenario, and user outcome in the same workflow.

A practical setup starts with simulate-sdk LiveKitEngine. The team runs scripted refund, scheduling, and safety-disclosure scenarios, captures caller audio and agent audio, and stores the intended response text, generated TTS audio, voice ID, locale, scenario ID, turn timestamps, interruption events, and call outcome. The nearest built-in evaluators are then attached as companion checks: AudioQualityEvaluator for speech artifacts, TTSAccuracy for whether generated speech matches intended content, and Tone for text-level tone conformance. Prosody-specific labels can be added as human annotations: pitch naturalness, stress placement, pause timing, emotional fit, and speaking-rate acceptability.

The next action is concrete. If a new TTS provider raises task completion but doubles repeat requests in Spanish billing calls, the engineer samples those traces, listens to the failing turns, compares TTSAccuracy and Tone results, and adds the clips to a regression dataset before rolling the route forward. Unlike Vapi-style transcript QA, this workflow does not treat readable text as proof that the spoken experience worked. It keeps audio, timing, evaluator output, and user reaction connected to the same trace so teams can debug the exact turn that sounded wrong.

How to Measure or Detect Prosody

Measure prosody as a cohort-aware set of delivery signals, not as one transcript score:

• Human prosody labels - rate pitch naturalness, stress placement, pause timing, speaking rate, emotional fit, and perceived empathy on sampled audio.
• AudioQualityEvaluator - checks nearby audio issues such as clipping, distortion, echo, or dropout that can mask prosody problems.
• TTSAccuracy - verifies that generated speech matches the intended text before delivery quality is judged.
• Tone - checks whether the response style matches the expected tone; pair it with audio review because tone text is not pitch contour.
• Dashboard signals - track barge-in rate, repeat-request rate, turn overlap, long silence, p99 time-to-first-audio, and eval-fail-rate-by-cohort.
• User-feedback proxies - use escalation rate, abandoned calls, post-call thumbs-down, and reopened tickets after a “resolved” voice call.

Minimal companion-eval pattern:

from fi.evals import AudioQualityEvaluator

result = AudioQualityEvaluator().evaluate(
    audio_path="calls/agent-turn.wav",
    metadata={"scenario": "refund", "locale": "en-US"}
)
print(result.score, result.reason)

Use the score to filter obviously bad audio, then review prosody labels on the remaining high-impact turns.

Common Mistakes

• Treating transcript correctness as delivery quality. A perfect transcript loses pitch, stress, rhythm, pause length, and whether the voice sounded dismissive.
• Using one TTS voice for every workflow. Collections, healthcare, onboarding, and outage messaging need different emotional range and pacing.
• Optimizing only for speed. Shortening pauses can reduce latency while making the agent interrupt, sound anxious, or rush required disclosures.
• Testing studio prompts only. Production calls include background speech, accents, partial barge-in, noisy phones, and emotionally loaded requests.
• Inferring prosody from ASR confidence. ASR can be confident on speech that sounds unnatural, poorly stressed, or inappropriate for the moment.