Articles

Chatbot Analytics From Conversation Logs to a Ranked Fix List

Chatbot analytics that finds what failed: sample transcripts by intent, measure resolution and drop-off, score each chat with an LLM judge, rank fixes.

· 21 min read
chatbot-analytics conversation-analytics llm-as-judge customer-support embeddings python
A three step podium ranking the dead end, escalation loop and wrong help article failure clusters by score, with conversation bubbles above it.

The weekly report on the support bot looks calm. Most conversations never reached a person, and the survey average sits above four out of five. Then finance forwards a stack of customer emails. People asked the bot for last month’s invoice, got nowhere, and wrote in instead. None of them appears anywhere in the report.

Chatbot analytics built on volume, containment and survey scores stops at a report like that one. It counts conversations and never outcomes, so nobody on the team can say which intent fails or why users leave. Containment goes up when a frustrated customer closes the tab, and the survey only hears from people who stayed long enough to answer it.

This guide turns raw conversation logs into a ranked fix list in six steps: sample transcripts by intent and outcome, label intents, measure resolution and drop-off per intent, score each conversation with an LLM judge, cluster the judge’s reasons, and rank them. A Python pipeline runs each step on a synthetic support-bot log, so you can reproduce every number below.

TL;DR

  • Volume, containment and CSAT count conversations and cannot show which users left without an answer.
  • Chatbot analytics you can act on reads and judges a stratified sample by intent and outcome, then weights it back to the full log.
  • Label intents with a classifier or a judge, check the labels against a hand-labelled slice, and report resolution, drop-off and escalation per intent.
  • An LLM judge scores each conversation and writes a reason, and embedding those reasons groups failures into recurring problems.
  • Rank each problem by weighted frequency times impact, fix the top one, and measure it again on a fresh sample.

Volume, containment and CSAT hide the conversations that failed

Volume counts conversations, containment counts the ones that never reached a person, and CSAT averages the few users who answered a survey. None of the three says whether the user got what they came for. A user who gives up and closes the chat counts as contained and leaves no score.

Every number in this guide comes from one synthetic log of 2,400 conversations with a made-up support bot for a SaaS workspace product, generated with a fixed seed. The bot handles six intents plus an other bucket, from inviting a teammate to fixing an SSO login. No real customer wrote any of these conversations.

On that log the dashboard reads well. Containment is 93.8%, and 83.4% of conversations end with the user confirming the fix. The gap between those two numbers is 251 abandoned conversations, 10.5% of the log, where the user left with neither an answer nor a handoff. Each of them still counts as contained.

That gap is why conversation analytics has to reach the outcome of each conversation. Containment tells you nothing about whether the user left with an invoice, a working login or a dead end, and those are the cases a support lead needs to see.

Goal conversion and user sentiment earn a place on the dashboard too, but reported as one total across every conversation, neither names the intent that failed. For a live view, the monitoring guide lists the metrics worth monitoring on every conversation.

If containment stays on your dashboard, the support bot guide shows how to pair it with a false resolution rate so a contained chat that should have escalated still shows up.

Table 1 maps each dashboard number to the users it leaves out and to the per-intent check that covers them. The sections below build those checks in order and end with a list of problems ranked by what they cost your users.

NumberWhat it countsWhat it missesThe per-intent check that replaces it
VolumeConversations startedWhether any were solvedResolution per intent
ContainmentConversations not handed to a personUsers who gave up inside the chatDrop-off per intent
CSATUsers who answered a surveyUsers who left before the surveyJudge score on every sampled conversation
Resolution overallConversations marked resolvedWhich intent drags it downResolution per intent with an interval

What the survey never hears

CSAT on the synthetic log came from 320 responses, 13.3% of conversations, with a mean of 4.19 out of 5. Only 2 of the 251 abandoned conversations left a response. On this log the survey mostly heard from users who got their answer, and the average reflects their experience.

A high average then says little about the users who left. Their view sits outside the number, so a survey trend can stay flat while one intent loses customers. Drop-off per intent and a judge score on each sampled conversation reach the people the survey misses, which the next sections set up.

Sampling transcripts by intent and outcome

Sampling by cell means grouping conversations by intent and outcome, reading up to a fixed number from each cell and weighting each transcript by the size of its cell. You still read the rare intents and the rare failures, and the weights put your overall failure rate back where it belongs.

A plain random sample mirrors traffic. The big, healthy intents eat most of the reading budget, a small intent such as api_rate_limit gets few reads, and its rare failures may not show up at all. You end up reading dozens of resolved invite_teammate chats and almost none of the chats that went wrong.

The pipeline groups the log into cells of intent times outcome, which gives 21 cells here, from seven intent labels and three outcomes. It reads at most 12 conversations per cell and takes every conversation from a smaller cell. That adds up to 249 conversations read, and each one keeps its cell and its weight.

def stratified_sample(rng, rows):
    cells = defaultdict(list)
    for r in rows:
        cells[(r["intent"], r["outcome"])].append(r)
    sample = []
    for key in sorted(cells):
        members = cells[key]
        picked = rng.sample(members, min(CELL_CAP, len(members)))
        for r in picked:
            r["weight"] = len(members) / len(picked)   # population conversations this one stands for
            r["cell"] = key
        sample.extend(picked)
    return sample

The weight is the number of conversations in the cell divided by the number read from it. If a cell holds 300 conversations and you read 12, each read stands for 25. If a cell holds 9 and you read all 9, each stands for 1. Weighted shares add those stand-in counts back up to the full log.

If the judge’s cost per call fits your budget, score the full log and the judge numbers need no weights. Keep the stratified sample anyway for the reading people do by hand, and as the cap on judge spend once traffic grows past what the budget covers.

How many conversations to read

The usual sample size for estimating a proportion is n = z^2 * p * (1 - p) / e^2, with z at 1.96 for 95% confidence and p at 0.5 as the worst case. The script prints 97 conversations for a margin of plus or minus 10 points, and 385 for plus or minus 5 points.

Those counts apply to what you read and judge. Outcome rates such as resolution come from log fields, so you compute them on every conversation and need no sample for them. The sample covers the work that costs a person’s time or a judge call, which is reading transcripts and scoring them.

Even the full log leaves each intent with a few hundred conversations, from 229 for cancel_plan to 433 for invite_teammate. At that size a single resolution rate carries real noise, so a two-point gap between intents can mean nothing. That is why the per-intent table further down puts an interval on every rate.

Weighting the sample back to the full log

Because the data is synthetic, the script knows the true share of conversations with a failure, 12.5%. The 249 conversations read, counted without weights, put it at 41.0%. With the stratum weights applied, the estimate drops to 10.9%, close to the true share and nowhere near the unweighted figure.

The unweighted number goes wrong because the cap over-reads the failing cells. Abandoned and escalated chats sit in small cells that get read at a far higher rate than large resolved cells, which get 12 reads no matter how big they are. A share computed straight from the sample inherits that tilt toward failure.

Report weighted shares every time you state a rate for the whole log. Keep each conversation’s cell label as well, because the re-measure step later resamples inside each cell to put an interval on the before and after numbers, and a plain bootstrap would treat the sample as random.

Intent labels you can check against a human

Write a short intent list with an other bucket, label every conversation with a cheap classifier or an LLM judge, then hand-label a slice and measure agreement. Report Cohen’s kappa beside raw agreement, because raw agreement looks high whenever one intent dominates, and kappa discounts the matches that label frequencies alone would produce.

Picture a labeller that tags every conversation with the most common intent. It agrees with a person on that whole share of traffic and learns nothing. Kappa compares observed agreement with the agreement two labellers would reach by guessing at their own label frequencies, so that lazy labeller scores close to zero.

Six intents and an other bucket

Start from what support already knows: the top ticket categories and the product areas users ask about. The synthetic bot uses invite_teammate, export_data, sso_login, invoice_copy, cancel_plan, api_rate_limit and other. Each conversation gets one label from its first user turn, since that turn says why the user opened the chat.

Keep the list under about ten labels plus other, and write a one-line description for each. The descriptions matter, because the classifier compares each opener against them. The description behind invoice_copy reads “get a copy of an invoice or receipt for billing”, written the way a user would ask.

Split an intent when its conversations need different fixes, and merge two when the bot handles them the same way. A request that keeps landing in the other bucket for weeks deserves its own label and its own description, and adding it shrinks the bucket where labels drift most.

Classifier or judge for intent labels

The pipeline’s classifier embeds the first user turn and every intent description with fastembed and the BAAI/bge-small-en-v1.5 model, which runs on your machine. It then picks the description closest to the opener. There is no API call and no cost per conversation, so labelling the full log takes one pass.

An LLM judge is the upgrade for long, rambling openers where the request sits three sentences in, or where one message mixes two problems. It reads the turn the way a person would, at the price of one model call per conversation. Start with the classifier and move to a judge for intents that fail the agreement check.

The agreement check on a hand-labelled slice

On a slice of 60 conversations the classifier matched the hand labels 93.3% of the time, and Cohen’s kappa came out at 0.92. In this synthetic run the hand labels are the generator’s own intent field, used as an approximation of a person’s read. With real logs, someone from support labels the slice.

All four misses came from the other bucket: two other conversations went to invoice_copy and two went to invite_teammate. Other holds openers that match no description well, such as “Hello?” or a partnership question, so a weak nearest match can pull them into a real intent. Check that bucket first each month.

Chatbot analytics per intent for resolution, drop-off and escalation

AI chatbot analytics should break resolution, drop-off and escalation out by intent and put an interval on each rate. A bot that resolves most conversations overall can still fail one intent badly, and a per-intent table shows which one. Use the Wilson interval, since small intents give wide ranges that a bare percentage hides.

Table 2 is the per-intent view of the full synthetic log of 2,400 conversations, sorted from the lowest resolution rate to the highest. The interval is a 95% Wilson interval on resolution, and the drop-off and escalation columns are plain shares of each intent’s conversations. The Wilson function is eight lines.

def wilson(k, n, z=1.96):
    if n == 0:
        return (0.0, 0.0)
    p = k / n
    d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return (c - h, c + h)
IntentConversationsResolved95% intervalDrop-offEscalated
invoice_copy32875.0%70.0% to 79.4%19.8%5.2%
sso_login38579.0%74.6% to 82.7%10.9%10.1%
export_data36382.4%78.1% to 85.9%9.6%8.0%
api_rate_limit23083.0%77.7% to 87.3%9.6%7.4%
cancel_plan22985.2%80.0% to 89.2%9.2%5.7%
invite_teammate43386.6%83.1% to 89.5%7.9%5.5%
other43290.5%87.4% to 92.9%7.4%2.1%

invoice_copy resolves least, at 75.0% with an interval of 70.0% to 79.4%, and it has the highest drop-off in the table at 19.8%. Users who ask for an invoice walk away without one almost twice as often as users of sso_login, the next highest at 10.9%. That row is the stack of emails finance forwarded.

sso_login escalates most, at 10.1%, and its drop-off sits close behind at 10.9%, so its failures split between a handoff and a user leaving. That calls for a different fix than invoice_copy needs. An escalation costs your support team time, but the user still reaches someone who can sort out the login.

Check the intervals first. sso_login runs from 74.6% to 82.7% and export_data from 78.1% to 85.9%, so they overlap and the gap between them may be noise. invoice_copy’s interval clears cancel_plan, invite_teammate and other but still overlaps three intents, and its drop-off is what sets it apart.

Defining resolved, dropped and escalated

Settle the definitions first. In this guide, resolved means the user confirmed the fix, dropped means the user left with neither a resolution nor a handoff, and escalated means the bot handed the chat to a person. Each outcome needs one signal in the log.

A confirmation turn such as “Thanks, that worked” marks resolved. A session that ends with no reply after the bot’s last turn marks dropped, and a handoff event marks escalated. Keep the same signals from month to month, or next month’s rates will not compare with this month’s.

Scoring each conversation with an LLM judge

For AI conversation analysis, an LLM judge reads one whole transcript against a written rubric and returns a score and a short reason. The score catches conversations that ended politely without helping. The reason feeds the clustering step, so ask the judge to name the single biggest thing the bot got wrong.

The rubric in the pipeline has three levels. A conversation scores 1.0 if the bot fully solved the problem, 0.5 if it partly solved it or solved it only after the user repeated themselves, and 0.0 if it did not. The reason is one plain sentence naming the most important thing the bot got wrong.

Score the whole conversation as one unit. A bot can give five reasonable replies and still leave the user without an invoice, and a turn-level average would hide that. Turn weighting and rubric wording get their own treatment in the guide on designing rubrics for multi-turn conversations.

The synthetic log carries judge_score and judge_reason fields that stand in for real judge output. Their scores are continuous values, while a live judge returns one of the three rubric levels. The clustering step flags anything at 0.5 or lower, so partly solved conversations get grouped along with outright failures.

Calling the LLM judge from Python

The judge() function calls evaluate() from fi.evals, which ships in the agent-learning-kit package, version 0.3.7 as of 2026-11-12. It passes the rubric as the prompt, engine=“llm”, a model name that LiteLLM routes to its provider, and the transcript as the conversation keyword argument.

RUBRIC = (
    "You grade one customer-support chatbot conversation. Score 1.0 if the user's "
    "problem was fully solved by the bot, 0.5 if partly solved or solved only after "
    "the user repeated themselves, 0.0 if it was not solved. In the reason, name the "
    "single most important thing the bot got wrong, in one plain sentence, or say "
    "what it did right."
)


def judge(transcript, model):
    """Score one conversation with an LLM judge through agent-learning-kit.

    # not executed: needs a provider key (LiteLLM routes `model` to its provider)
    The inputs are passed as keyword arguments and are dumped to JSON under INPUT in the
    judge prompt, so the rubric does not use {placeholders}.
    A real judgement comes back as a score plus a reason, and the reason is the judge's
    whole JSON reply ({"score": ..., "reason": ...}), so the reason field is pulled out.
    The code stops when the status is not completed or when the reason is not the
    judge's JSON reply, so no error text reaches the clusters.
    """
    from fi.evals import evaluate
    res = evaluate(prompt=RUBRIC, engine="llm", model=model, conversation=transcript)
    if res.status != "completed":
        raise RuntimeError(f"judge failed: {res.error}")
    try:
        reply = json.loads(res.reason)
    except ValueError:
        raise RuntimeError(f"judge failed: {res.reason}") from None
    return res.score, reply.get("reason", "")

The code stops with an error when the result status is not completed or when the reason is not the judge’s JSON reply, so no error text reaches the clusters. This call was not run for this guide because it needs a provider key. The script prints the end of the prompt the judge would see.

== 5. Judge step
not executed: needs a provider key. Tail of the prompt the judge would receive for one conversation:
the following input.

### INPUT ###
{
  "conversation": "user: I keep getting 429 errors from the API.\nbot: Yes, that is included. Open Settings and turn it on.\nuser: That did not help."
}
JUDGE_MODEL not set: the steps below use the synthetic judge_score and judge_reason fields

What a judge cannot catch

A judge only sees the transcript. It cannot tell whether the plan limit the bot quoted was true for that account, and it cannot see what happened after the chat, such as a repeat contact the next day or a reopened ticket. Those checks need account data and ticket history joined to the conversation.

Judges also drift when you change the model or the rubric. Spot-check judge scores against human reads on the same hand-labelled slice you used for intents, and repeat the check after any change to either. A judge that keeps disagreeing with your support team on that slice is grading something other than what you meant.

Clustering failure reasons into recurring problems

Clustering starts from every conversation the judge scored 0.5 or lower: embed its reason with a local embedding model and group reasons that sit close together. Each group is one recurring problem in plain words. Read a few members of each group before you name it, because one cluster can mix two causes.

Funnel from 2,400 synthetic conversation logs to 249 sampled conversations, 102 flagged by the judge and 12 clusters, feeding a ranked list of six fixes with their scores

Cluster the judge’s reasons and leave the transcripts out of the embedding step. A transcript holds greetings, account details and small talk, while the reason is already the judge’s diagnosis in one sentence. Two reasons that both describe a dead end in different words land near each other, even when the transcripts behind them look nothing alike.

The pipeline uses average-linkage clustering on cosine distance, written in numpy. It starts with every reason in its own group and keeps merging the two closest groups until the closest pair sits farther apart than a cutoff. For a few hundred reasons that loop runs fine on a laptop.

def agglomerate(vecs, max_dist):
    """Average-linkage clustering on cosine distance. O(n^3), fine for a few hundred reasons.
    Swap for scikit-learn AgglomerativeClustering past a few thousand items.
    """
    clusters = [[i] for i in range(len(vecs))]
    dist = 1 - vecs @ vecs.T
    while len(clusters) > 1:
        best, pair = 9.0, None
        for a in range(len(clusters)):
            for b in range(a + 1, len(clusters)):
                d = dist[np.ix_(clusters[a], clusters[b])].mean()
                if d < best:
                    best, pair = d, (a, b)
        if best > max_dist:
            break
        a, b = pair
        clusters[a] += clusters.pop(b)
    return clusters

In the synthetic run the judge fields flagged 102 of the 249 sampled conversations, and 12 clusters ended up with 3 or more members. Groups under 3 members go to a watch list that stays off the fix list. Check the watch list next month, and promote a group once it grows past that size.

Name each cluster from its most central reason, the member closest to the cluster’s average vector. Then read four or five member transcripts before you write the name on the fix list. Two causes can sound alike in a one-sentence reason and still need different fixes, and only the transcripts tell them apart.

For trace-level clustering of agent runs, read debugging agent failures from traces, since this guide stays on support and product conversations.

Local embeddings for failure reasons

The pipeline embeds text with fastembed and the BAAI/bge-small-en-v1.5 model, which run on a laptop with no API key. The first run downloads the model files from Hugging Face, and later runs reuse them. The same embeddings label intents in the classifier step.

_model = None


def embed(texts):
    global _model
    if _model is None:
        from fastembed import TextEmbedding
        _model = TextEmbedding(model_name=EMBED_MODEL)
    v = np.array(list(_model.embed(list(texts))), dtype=np.float64)
    return v / np.linalg.norm(v, axis=1, keepdims=True)

The embed() function scales every vector to unit length, so a dot product between two vectors equals their cosine similarity. That keeps both steps to plain matrix products in numpy: one product scores openers against intent descriptions, and another builds the distance matrix the clustering loop reads.

Ranking chatbot analytics failures by frequency times impact

Each cluster gets a score of weighted frequency times impact. Frequency is the share of all conversations the cluster stands for once you apply the sampling weights. Impact is the average cost of its outcomes: an abandoned chat costs more than an escalated one, and a flagged chat the user still finished costs least.

In code, frequency is the summed weight of a cluster’s members divided by the summed weight of the whole sample, and impact is the weighted average cost of those members’ outcomes. Multiply the two and each cluster gets one number that anyone on the team can check against the printed output.

The impact weights are a choice, made for this sample: abandoned 1.0, escalated 0.5, flagged but resolved 0.25. An abandoned user got nothing, an escalated user still reached a person, and a flagged user who finished paid with extra turns. Set the weights to what each outcome costs your team, and record them beside the ranking.

== 7. Ranked fix list: score = weighted frequency x impact
impact weights: abandoned 1.0, escalated 0.5, flagged but resolved 0.25 (chosen for this sample)
#1 score 0.0204 | freq 2.3% | impact 0.90 | 16 read | top intents [('invoice_copy', 16)]
    example reason: The bot said it could not find the item and gave no next step.
#2 score 0.0162 | freq 2.0% | impact 0.81 | 20 read | top intents [('cancel_plan', 10), ('sso_login', 10)]
    example reason: Bot refused or ignored repeated requests to talk to a person.
#3 score 0.0092 | freq 1.1% | impact 0.84 | 9 read | top intents [('sso_login', 4), ('other', 3)]
    example reason: The bot linked a help article about a different feature than the one asked about.
#4 score 0.0079 | freq 1.1% | impact 0.69 | 13 read | top intents [('export_data', 7), ('api_rate_limit', 6)]
    example reason: Bot stated a limit that does not match the user's plan.
#5 score 0.0077 | freq 0.9% | impact 0.90 | 9 read | top intents [('export_data', 5), ('api_rate_limit', 4)]
    example reason: The reply described a capability the account does not include.
#6 score 0.0044 | freq 0.5% | impact 0.84 | 5 read | top intents [('invite_teammate', 5)]
    example reason: Steps in the answer do not match the current interface.

The dead end leads with a score of 0.0204, from 2.3% of conversations, an impact of 0.90, and all 16 sampled members in invoice_copy. Its central reason reads “The bot said it could not find the item and gave no next step.” That is the invoice problem from finance’s inbox, now with a size.

Second is the escalation loop at 0.0162, covering 2.0% of conversations, split evenly between cancel_plan and sso_login. Users asked for a person and the bot kept answering. Its impact of 0.81 sits nearer the 1.0 of an abandoned chat than the 0.5 of an escalation, so most of them left without that person.

On a plot of frequency against impact, the top two sit apart from the rest on frequency. A wrong help article scores 0.0092, a limit that does not match the user’s plan 0.0079, a capability the account does not include 0.0077, and steps that no longer match the interface 0.0044.

Two by two grid of weighted frequency against impact with the dead end and escalation loop clusters in the high frequency, high impact quadrant and the other four clusters on the low frequency side

Clusters #4 and #5 describe one cause in two wordings: the bot claimed something about the user’s plan that was not true. Both draw from export_data and api_rate_limit, and their scores add up to 0.0156, enough to rank third. Merge them by hand after reading the members, and leave the cutoff alone.

Give each item an owner and the place where its fix goes. The dead end and the escalation loop need a handoff rule, the wrong article needs help content, stale steps need product docs, and the plan claims need a prompt change.

Re-measuring after a fix

In the default run, the pipeline builds a second synthetic log where dead-end failures happen at 20% of their old rate, simulating a shipped handoff. It samples that log the same way and assigns each new flagged reason to the nearest old cluster centroid when it sits close enough, so before and after count the same problem.

A stratified bootstrap puts an interval on the share. It resamples inside each cell, but scales each replicate’s deviation by sqrt(1 - n/N), so a cell you read in full contributes no sampling uncertainty and a cell you sampled still does. Before the fix the #1 cluster covered 2.3% of conversations (95% interval 1.6% to 2.9%). After it, the share was 0.2% (0.2% to 0.4%), and invoice_copy resolution on the full log rose from 75.0% to 90.8%.

Call a fix done when the before and after intervals do not overlap, as here. If they overlap, read more conversations from the affected cells or wait for more traffic before you claim the win. Then rerun the ranking and see which cluster now leads your list.

Running the pipeline on the synthetic log

Every step except the judge call was tested end to end on Python 3.12 with numpy 2.5.4, fastembed 0.9.1 and agent-learning-kit 0.3.7. Save the file below as pipeline.py and run the uv command under it. With the seed fixed at 7 and JUDGE_MODEL unset, every run prints the same output.

The data is synthetic, and it cannot show how real users talk. They ramble, switch topics halfway through a chat and paste error logs, and the generated openers do none of that. Set JUDGE_MODEL and that provider’s key to judge the baseline sample instead of using its synthetic judge fields.

That optional branch is baseline-only: it clusters and ranks the live judge’s results on the synthetic transcripts, then stops. It does not compare them with a synthetically scored after sample. A real before/after study needs distinct exports scored with the same judge, model and rubric.

"""Chatbot analytics pipeline: conversation logs to a ranked fix list.

All data here is SYNTHETIC: a made-up SaaS workspace support bot, generated with a
fixed seed. Every number this script prints comes from that synthetic data.

Steps
 1. containment, resolution and CSAT as a dashboard sees them
 2. sample size arithmetic and a stratified sample (intent x outcome)
 3. intent labelling with local embeddings, agreement with a hand-labelled slice
 4. resolution, drop-off and escalation per intent
 5. per-conversation judge step (needs a provider key, not executed here)
 6. failure-reason clustering with local embeddings
 7. ranking: weighted frequency x impact
 8. re-measure after a fix, with a bootstrap interval (synthetic judge mode only)
"""

import json
import math
import os
os.environ.setdefault("HF_HUB_DISABLE_PROGRESS_BARS", "1")
import random
from collections import Counter, defaultdict

import numpy as np

SEED = 7
N_POP = 2400            # conversations in the synthetic log
CELL_CAP = 12           # most conversations read per (intent, outcome) cell
EMBED_MODEL = "BAAI/bge-small-en-v1.5"

# ---------------------------------------------------------------- synthetic data
INTENTS = {  # intent -> (share of traffic, description used by the classifier)
    "invite_teammate": (0.18, "invite a teammate or add a member to my workspace"),
    "export_data": (0.15, "export my data or download a report as a file"),
    "sso_login": (0.17, "single sign-on login problem, cannot sign in with SSO"),
    "invoice_copy": (0.14, "get a copy of an invoice or receipt for billing"),
    "cancel_plan": (0.10, "cancel my subscription or downgrade my plan"),
    "api_rate_limit": (0.08, "API rate limit errors and request quotas"),
    "other": (0.18, "general question that fits none of the other topics"),
}
OPENERS = {
    "invite_teammate": ["How do I add my colleague to our workspace?", "I need to invite a new teammate, where is the button?", "Can I send an invite to someone on my team?"],
    "export_data": ["I want to download all my project data.", "How can I export a report to CSV?", "Need to get my data out as a file."],
    "sso_login": ["Our team cannot log in with SSO since this morning.", "SSO sign in keeps sending me back to the login page.", "Single sign-on is failing for one of our users."],
    "invoice_copy": ["Can you send me last month's invoice?", "I need a receipt for my accountant.", "Where do I find a copy of our invoice?"],
    "cancel_plan": ["I want to cancel my plan.", "How do I downgrade to the free tier?", "Please cancel our subscription."],
    "api_rate_limit": ["I keep getting 429 errors from the API.", "What is our API request limit per minute?", "Rate limit hit again, can we raise it?"],
    "other": ["Do you have a mobile app?", "Is there a student discount?", "Who do I talk to about a partnership?", "Hello?"],
}
# failure type -> (judge-style reasons, bot turn, outcome probabilities resolved/abandoned/escalated)
FAILURES = {
    "wrong_article": (
        ["The bot linked a help article about a different feature than the one asked about.",
         "Answer pointed to an unrelated help page, the user's question stayed unanswered.",
         "The linked article does not cover the problem the user described.",
         "Bot returned a help page for the wrong topic and offered nothing else."],
        "Here is an article that might help: Getting started with dashboards.",
        (0.15, 0.60, 0.25)),
    "repeat_question": (
        ["The bot asked for the account email twice after the user had already given it.",
         "User had to repeat the same detail, the bot asked for it again.",
         "Bot re-asked for information already provided earlier in the chat.",
         "The same clarifying question appeared twice in a row."],
        "Could you confirm your account email again?",
        (0.20, 0.55, 0.25)),
    "dead_end": (
        ["The bot said it could not find the item and gave no next step.",
         "Conversation ended on 'I could not find that' with no handoff or alternative.",
         "No result was found and the bot offered no way forward.",
         "Bot gave up after one failed lookup and did not offer a human."],
        "I could not find that. Is there anything else I can help with?",
        (0.05, 0.80, 0.15)),
    "wrong_plan_claim": (
        ["The bot promised a feature that is not on the user's plan.",
         "Answer assumed a higher plan tier than the account has.",
         "Bot stated a limit that does not match the user's plan.",
         "The reply described a capability the account does not include."],
        "Yes, that is included. Open Settings and turn it on.",
        (0.10, 0.45, 0.45)),
    "stale_steps": (
        ["The bot described menu steps that no longer exist in the product.",
         "Instructions referenced a button that has been renamed, the user got stuck.",
         "Out-of-date navigation steps were given.",
         "Steps in the answer do not match the current interface."],
        "Go to Settings, then Team, then press the blue Add button.",
        (0.40, 0.40, 0.20)),
    "escalation_loop": (
        ["The user asked for a human three times and the bot kept answering itself.",
         "Bot refused or ignored repeated requests to talk to a person.",
         "Handoff to a human was asked for and not offered.",
         "The bot looped on its own answer after the user asked for an agent."],
        "I can help with that. Could you tell me more?",
        (0.05, 0.50, 0.45)),
}
# intent -> list of (failure type, probability a conversation of this intent has it)
FAILURE_MAP = {
    "invite_teammate": [("stale_steps", 0.10), ("wrong_article", 0.04)],
    "export_data": [("wrong_plan_claim", 0.12), ("stale_steps", 0.05)],
    "sso_login": [("escalation_loop", 0.10), ("wrong_article", 0.06), ("repeat_question", 0.04)],
    "invoice_copy": [("dead_end", 0.22)],
    "cancel_plan": [("repeat_question", 0.08), ("escalation_loop", 0.05)],
    "api_rate_limit": [("wrong_plan_claim", 0.15)],
    "other": [("wrong_article", 0.03)],
}
CLEAN_OUTCOME = (0.93, 0.04, 0.03)
OUTCOMES = ("resolved", "abandoned", "escalated")
COST = {"resolved": 0.0, "escalated": 0.5, "abandoned": 1.0}  # weights chosen for this sample
FLAGGED_RESOLVED_COST = 0.25   # judge flagged it, user still finished the task
CSAT_RESPONSE = {"resolved": 0.15, "escalated": 0.10, "abandoned": 0.02}  # synthetic assumption
CSAT_MEAN = {"resolved": 4.4, "escalated": 3.2, "abandoned": 2.0}


def make_log(rng, n, fixed=None):
    """Return n synthetic conversations. `fixed` maps failure type -> multiplier (a shipped fix)."""
    fixed = fixed or {}
    names = list(INTENTS)
    weights = [INTENTS[i][0] for i in names]
    rows = []
    for k in range(n):
        intent = rng.choices(names, weights)[0]
        failure = None
        for ftype, p in FAILURE_MAP[intent]:
            if rng.random() < p * fixed.get(ftype, 1.0):
                failure = ftype
                break
        probs = FAILURES[failure][2] if failure else CLEAN_OUTCOME
        outcome = rng.choices(OUTCOMES, probs)[0]
        opener = rng.choice(OPENERS[intent])
        turns = [("user", opener)]
        if failure:
            turns += [("bot", FAILURES[failure][1]), ("user", "That did not help.")]
            reason = rng.choice(FAILURES[failure][0])
            score = round(rng.uniform(0.05, 0.4), 2)
        else:
            turns += [("bot", "Done, here are the steps. Did that solve it?")]
            reason = ""
            score = round(rng.uniform(0.75, 1.0), 2)
        if outcome == "resolved":
            turns.append(("user", "Thanks, that worked."))
        csat = None
        if rng.random() < CSAT_RESPONSE[outcome]:
            csat = min(5, max(1, round(rng.gauss(CSAT_MEAN[outcome], 0.8))))
        rows.append({"id": k, "intent": intent, "outcome": outcome, "true_failure": failure,
                     "transcript": "\n".join(f"{r}: {t}" for r, t in turns),
                     "opener": opener, "judge_score": score, "judge_reason": reason, "csat": csat})
    return rows


# ---------------------------------------------------------------- statistics
def wilson(k, n, z=1.96):
    if n == 0:
        return (0.0, 0.0)
    p = k / n
    d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return (c - h, c + h)


def cohen_kappa(a, b):
    n = len(a)
    po = sum(x == y for x, y in zip(a, b)) / n
    ca, cb = Counter(a), Counter(b)
    pe = sum(ca[l] * cb[l] for l in set(a) | set(b)) / (n * n)
    return (po - pe) / (1 - pe)


def stratified_sample(rng, rows):
    cells = defaultdict(list)
    for r in rows:
        cells[(r["intent"], r["outcome"])].append(r)
    sample = []
    for key in sorted(cells):
        members = cells[key]
        picked = rng.sample(members, min(CELL_CAP, len(members)))
        for r in picked:
            r["weight"] = len(members) / len(picked)   # population conversations this one stands for
            r["cell"] = key
        sample.extend(picked)
    return sample


# ---------------------------------------------------------------- embeddings
_model = None


def embed(texts):
    global _model
    if _model is None:
        from fastembed import TextEmbedding
        _model = TextEmbedding(model_name=EMBED_MODEL)
    v = np.array(list(_model.embed(list(texts))), dtype=np.float64)
    return v / np.linalg.norm(v, axis=1, keepdims=True)


def agglomerate(vecs, max_dist):
    """Average-linkage clustering on cosine distance. O(n^3), fine for a few hundred reasons.
    Swap for scikit-learn AgglomerativeClustering past a few thousand items.
    """
    clusters = [[i] for i in range(len(vecs))]
    dist = 1 - vecs @ vecs.T
    while len(clusters) > 1:
        best, pair = 9.0, None
        for a in range(len(clusters)):
            for b in range(a + 1, len(clusters)):
                d = dist[np.ix_(clusters[a], clusters[b])].mean()
                if d < best:
                    best, pair = d, (a, b)
        if best > max_dist:
            break
        a, b = pair
        clusters[a] += clusters.pop(b)
    return clusters


# ---------------------------------------------------------------- judge step
RUBRIC = (
    "You grade one customer-support chatbot conversation. Score 1.0 if the user's "
    "problem was fully solved by the bot, 0.5 if partly solved or solved only after "
    "the user repeated themselves, 0.0 if it was not solved. In the reason, name the "
    "single most important thing the bot got wrong, in one plain sentence, or say "
    "what it did right."
)


def judge(transcript, model):
    """Score one conversation with an LLM judge through agent-learning-kit.

    # not executed: needs a provider key (LiteLLM routes `model` to its provider)
    The inputs are passed as keyword arguments and are dumped to JSON under INPUT in the
    judge prompt, so the rubric does not use {placeholders}.
    A real judgement comes back as a score plus a reason, and the reason is the judge's
    whole JSON reply ({"score": ..., "reason": ...}), so the reason field is pulled out.
    The code stops when the status is not completed or when the reason is not the
    judge's JSON reply, so no error text reaches the clusters.
    """
    from fi.evals import evaluate
    res = evaluate(prompt=RUBRIC, engine="llm", model=model, conversation=transcript)
    if res.status != "completed":
        raise RuntimeError(f"judge failed: {res.error}")
    try:
        reply = json.loads(res.reason)
    except ValueError:
        raise RuntimeError(f"judge failed: {res.reason}") from None
    return res.score, reply.get("reason", "")


def render_judge_prompt(transcript):
    """Print helper: build the message the judge receives, without calling any model.
    Uses library internals; the model name is only needed to construct the judge object."""
    from fi.evals.llm.providers.litellm import LiteLLMProvider
    from fi.evals.metrics.llm_as_judges.custom_judge.metric import CustomLLMJudge
    from fi.evals.metrics.llm_as_judges.types import CustomInput
    j = CustomLLMJudge(provider=LiteLLMProvider(), config={"grading_criteria": RUBRIC, "model": "gpt-4o"})
    return j._create_prompt_messages(CustomInput(conversation=transcript))[1]["content"]


# ---------------------------------------------------------------- analysis helpers
def weighted_share(sample, pred):
    tot = sum(r["weight"] for r in sample)
    return sum(r["weight"] for r in sample if pred(r)) / tot


def bootstrap_ci(rng, sample, pred, reps=1000):
    """Stratified bootstrap of a weighted share over a finite log.

    The weighted share is sum_h N_h * p_h / N, so each cell's rate p_h is resampled
    on its own. Each replicate's deviation is scaled by sqrt(1 - n_h / N_h): a cell
    read in full has no sampling uncertainty and is held fixed, while a cell read
    in part stays close to an ordinary bootstrap.
    """
    cells = defaultdict(list)
    for r in sample:
        cells[r["cell"]].append(r)
    strata = []
    for members in cells.values():
        n = len(members)
        pop = members[0]["weight"] * n                 # conversations in the cell
        p = sum(bool(pred(r)) for r in members) / n
        strata.append((pop, n, p, math.sqrt(max(0.0, 1.0 - n / pop)), members))
    total = sum(s[0] for s in strata)
    vals = []
    for _ in range(reps):
        num = 0.0
        for pop, n, p, fpc, members in strata:
            if fpc == 0.0:                             # cell read in full
                num += pop * p
                continue
            boot = sum(bool(pred(rng.choice(members))) for _ in range(n)) / n
            num += pop * (p + fpc * (boot - p))
        vals.append(num / total)
    vals.sort()
    return vals[int(0.025 * reps)], vals[int(0.975 * reps) - 1]


def cluster_flagged(flagged, max_dist):
    """Embed each judge reason and cluster. Returns clusters with members, centroid and vectors."""
    if not flagged:
        return []
    vecs = embed([r["judge_reason"] for r in flagged])
    out = []
    for g in agglomerate(vecs, max_dist):
        cen = vecs[g].mean(axis=0)
        cen /= np.linalg.norm(cen)
        out.append({"members": [flagged[i] for i in g], "vecs": vecs[g], "centroid": cen})
    return out


def assign(flagged, centroids, min_sim=0.78):
    """Nearest existing cluster for each reason, or -1 when nothing is close."""
    sims = embed([r["judge_reason"] for r in flagged]) @ centroids.T
    return [int(s.argmax()) if s.max() >= min_sim else -1 for s in sims]


def main():
    rng = random.Random(SEED)
    print("SYNTHETIC DATA: made-up SaaS support bot, seed", SEED, "- no real customer conversations")
    print("Python packages: numpy, fastembed (embedding model %s)\n" % EMBED_MODEL)
    log = make_log(rng, N_POP)

    # 1 the dashboard view
    n = len(log)
    contained = sum(r["outcome"] != "escalated" for r in log) / n
    resolved = sum(r["outcome"] == "resolved" for r in log) / n
    abandoned = sum(r["outcome"] == "abandoned" for r in log) / n
    answered = [r for r in log if r["csat"] is not None]
    n_ab = sum(r["outcome"] == "abandoned" for r in log)
    ab_with_csat = sum(r["csat"] is not None for r in log if r["outcome"] == "abandoned")
    print("== 1. What the dashboard shows (population of %d conversations)" % n)
    print(f"volume {n} | containment (not escalated) {contained:.1%} | resolved {resolved:.1%} | abandoned {abandoned:.1%}")
    print(f"CSAT: {len(answered)} responses ({len(answered)/n:.1%} of conversations), mean {np.mean([r['csat'] for r in answered]):.2f} of 5")
    print(f"abandoned conversations that left a CSAT response: {ab_with_csat} of {n_ab}")
    print(f"containment minus resolved = {contained - resolved:.1%} of conversations counted as contained but not resolved\n")

    # 2 sampling
    print("== 2. Sample size and stratified sample")
    for e in (0.10, 0.05):
        print(f"worst-case n for a proportion at +/-{e:.0%} (95% confidence): {math.ceil(1.96**2 * 0.25 / e**2)}")
    sample = stratified_sample(rng, log)
    print(f"cells (intent x outcome): {len({r['cell'] for r in sample})}, cap {CELL_CAP} per cell, conversations read: {len(sample)}")
    has_f = lambda r: r["true_failure"] is not None
    print(f"share of conversations with a failure: true {sum(map(has_f, log))/n:.1%} (known only because the data is synthetic), "
          f"unweighted sample {sum(map(has_f, sample))/len(sample):.1%}, "
          f"stratum-weighted sample {weighted_share(sample, has_f):.1%}\n")

    # 3 intent labels
    print("== 3. Intent labels: classifier vs hand-labelled slice (hand labels are synthetic)")
    names = list(INTENTS)
    cen = embed([INTENTS[i][1] for i in names])
    sl = rng.sample(sample, 60)
    pred = [names[int(k)] for k in (embed([r["opener"] for r in sl]) @ cen.T).argmax(axis=1)]
    hand = [r["intent"] for r in sl]   # stands in for hand labels (synthetic)
    acc = sum(a == b for a, b in zip(pred, hand)) / len(sl)
    print(f"slice of {len(sl)}: raw agreement {acc:.1%}, Cohen's kappa {cohen_kappa(hand, pred):.2f}")
    miss = Counter((h, p) for h, p in zip(hand, pred) if h != p)
    print("most common disagreements (hand -> classifier):", miss.most_common(3), "\n")

    # 4 per-intent rates from the log
    print("== 4. Per intent, from the full log (Wilson 95% interval on resolution)")
    print(f"{'intent':<16}{'n':>5}{'resolved':>10}  {'95% interval':<16}{'drop-off':>9}{'escalated':>11}")
    by = defaultdict(list)
    for r in log:
        by[r["intent"]].append(r)
    for i in sorted(by, key=lambda i: sum(r["outcome"] == "resolved" for r in by[i]) / len(by[i])):
        rs = by[i]
        k = sum(r["outcome"] == "resolved" for r in rs)
        lo, hi = wilson(k, len(rs))
        ab = sum(r["outcome"] == "abandoned" for r in rs) / len(rs)
        es = sum(r["outcome"] == "escalated" for r in rs) / len(rs)
        print(f"{i:<16}{len(rs):>5}{k/len(rs):>10.1%}  {f'[{lo:.1%}, {hi:.1%}]':<16}{ab:>9.1%}{es:>11.1%}")
    print()

    # 5 judge step
    print("== 5. Judge step")
    model = os.environ.get("JUDGE_MODEL")
    if model:
        print("JUDGE_MODEL set: baseline-only live judging; synthetic re-measurement is disabled")
        for r in sample:   # live path, only when JUDGE_MODEL and its provider key are set
            r["judge_score"], r["judge_reason"] = judge(r["transcript"], model)
    else:
        print("not executed: needs a provider key. Tail of the prompt the judge would receive for one conversation:")
        print(render_judge_prompt(sample[0]["transcript"])[-190:].strip())
        print("JUDGE_MODEL not set: the steps below use the synthetic judge_score and judge_reason fields\n")

    # 6 clustering
    print("== 6. Failure reasons clustered (flagged = judge score 0.5 or lower)")
    flagged = [r for r in sample if r["judge_score"] <= 0.5]
    clusters = [c for c in cluster_flagged(flagged, max_dist=0.22) if len(c["members"]) >= 3]
    print(f"flagged conversations in the sample: {len(flagged)}, clusters with 3 or more members: {len(clusters)}\n")

    # 7 ranking
    print("== 7. Ranked fix list: score = weighted frequency x impact")
    pop_n = sum(r["weight"] for r in sample)
    rows = []
    for c in clusters:
        ms = c["members"]
        w = sum(r["weight"] for r in ms)
        cost = lambda r: COST[r["outcome"]] if r["outcome"] != "resolved" else FLAGGED_RESOLVED_COST
        freq, impact = w / pop_n, sum(r["weight"] * cost(r) for r in ms) / w
        rep = ms[int((c["vecs"] @ c["centroid"]).argmax())]["judge_reason"]
        rows.append({"c": c, "score": freq * impact, "freq": freq, "impact": impact, "rep": rep,
                     "cause": None if model else Counter(r["true_failure"] for r in ms).most_common(1)[0][0],
                     "intents": Counter(r["intent"] for r in ms).most_common(2)})
    rows.sort(key=lambda x: -x["score"])
    print("impact weights: abandoned 1.0, escalated 0.5, flagged but resolved 0.25 (chosen for this sample)")
    for rank, x in enumerate(rows[:6], 1):
        print(f"#{rank} score {x['score']:.4f} | freq {x['freq']:.1%} | impact {x['impact']:.2f} | "
              f"{len(x['c']['members'])} read | top intents {x['intents']}")
        print(f"    example reason: {x['rep']}")
    print()

    if model:
        print("Baseline-only: no before/after comparison with synthetic judge fields.")
        return

    # 8 synthetic-only re-measurement after a fix to the first-ranked cluster
    top = rows[0]
    assert top["cause"] == "dead_end", f"the fix below targets dead_end, the ranking put {top['cause']} first"
    print("== 8. Re-measure after a fix to the #1 cluster (second synthetic log, dead_end failures cut to 20% of before)")
    rng2 = random.Random(SEED + 1)
    log2 = make_log(rng2, N_POP, fixed={"dead_end": 0.2})
    sample2 = stratified_sample(rng2, log2)
    cents = np.array([x["c"]["centroid"] for x in rows])
    for s in (sample, sample2):
        fl = [r for r in s if r["judge_score"] <= 0.5]
        for r, a in zip(fl, assign(fl, cents)):
            r["in_top"] = a == 0
    in_top = lambda r: r.get("in_top", False)
    for label, s in (("before", sample), ("after", sample2)):
        sh = weighted_share(s, in_top)
        lo, hi = bootstrap_ci(random.Random(SEED + 2), s, in_top)
        print(f"{label}: #1 cluster is {sh:.1%} of conversations (95% bootstrap interval {lo:.1%} to {hi:.1%})")
    rate = lambda lg: sum(r["outcome"] == "resolved" for r in lg if r["intent"] == "invoice_copy") / sum(r["intent"] == "invoice_copy" for r in lg)
    print(f"invoice_copy resolution from the full log, before to after: {rate(log):.1%} to {rate(log2):.1%}")


if __name__ == "__main__":
    main()
uv run --no-project --python 3.12 --with numpy==2.5.4 --with fastembed==0.9.1 --with agent-learning-kit==0.3.7 python -I pipeline.py

Scoring production conversations and grouping failures in Future AGI

In Future AGI Observe, evals score your production traces at the span, trace or session level, and a session is one whole conversation. Error Feed, available on Future AGI Cloud or an Enterprise deployment, groups failing traces that went wrong the same way into a single issue with event, trace and user counts.

Attach a judge such as the conversation_resolution eval to an eval task at session level. It is an LLM-as-a-judge eval that reads the full conversation and returns a score and a reason, the same pair the pipeline clusters. The task runs on historical traffic or continuously.

The steps to set up evals on Observe traces cover choosing the level and the run type.

The trace table shows the score in a column, which lets you filter down to the failed conversations and then set alerts on that score. That moves the per-conversation judging in this guide from a CSV export onto live sessions.

How Error Feed groups failures into issues starts with a sample of an Observe project’s traces. Traces that failed the same way become one issue with a fix layer, a severity and a status. A failing result from an eval task, with a reason, becomes an issue grouped by that eval.

Your frequency times impact score stays your own ranking: the issue counts show how often a problem fired and how many users it reached, and your impact weights decide the order. The platform code is at Future AGI on GitHub.

One conversation, one session

Give every trace of a conversation the same session.id. Set it once around each turn’s work and traceAI’s instrumentors put it on their spans, so Observe rebuilds the whole conversation from its traces. The session view is where a conversation-level eval reads the full transcript, as sessions in Observe explains.

Getting invoice copies off the top of the fix list

Customers asked the bot for an invoice and gave up. The per-intent table points at invoice copies, the judge reasons say the bot gave up with no next step, and the fix list puts that problem first. Add the handoff, sample again, and check that the cluster shrinks.

In the synthetic run the dead-end cluster fell from 2.3% of conversations to 0.2%, and invoice_copy resolution rose from 75.0% to 90.8%. The emails finance forwarded were a symptom of a failure the weekly report had no column for, and now it has one.

Use this synthetic pipeline as a template, not a ready-made real-log loader. Remove the generator-only truth checks and dead_end assertion, connect your exports and judge, and supply your own intents and descriptions. For re-measurement, use distinct before and after exports with the same judge, model and rubric. Read the top three clusters yourself before anyone ships a fix.

Frequently Asked Questions

What is conversational analytics?

Conversational analytics means reading chat and voice transcripts at scale to learn what users asked, whether they got it and where they gave up. Chatbot analytics applies the same idea to a bot: intents, resolution, drop-off, escalation and the recurring reasons conversations fail.

What should a chatbot analytics dashboard show?

A chatbot analytics dashboard should show resolution, drop-off and escalation for each intent, with an interval on every rate, next to volume and containment. Add the top recurring failure reasons from judged transcripts, ranked by how often they happen and what they cost the user.

How do you find chatbot failure modes and drop-offs?

Find drop-offs by logging every conversation with its intent, its outcome and a session id, then breaking drop-off out per intent. Find failure modes by having an LLM judge score a stratified sample by intent and outcome and explain each failure, then weight the results back to the full log before you compare intents.

How do embeddings power AI conversation analytics?

Embeddings turn each user message or judge reason into a vector, so text with a similar meaning lands close together even when the wording differs. They label intents by nearest description and group failure reasons into clusters you can rank.

How do containment and resolution rate differ in chatbot analytics?

Containment rate counts conversations that never reached a person, while resolution rate counts conversations where the user confirmed the problem was solved. A user who gives up and closes the chat raises containment but not resolution. In this guide's synthetic log of 2,400 conversations, containment was 93.8% and resolution 83.4%.
Related Articles
View all