What Is the Self-Consistency Evaluation Metric?

What Is the Self-Consistency Evaluation Metric?

The self-consistency evaluation metric measures how often a model returns the same answer when asked the same question repeatedly — usually with non-zero temperature, paraphrased prompts, or shuffled examples. A high score means the model is stable; a low score means the output is dominated by sampling noise rather than reasoning. Self-consistency is reference-free: it does not need a gold answer, which makes it cheap to attach to live production traces and useful as an early-warning signal for brittle prompts, ambiguous instructions, or under-specified rubrics.

Why It Matters in Production LLM and Agent Systems

A model that flips its answer between runs is a model you cannot ship. The pain shows up in customer-facing inconsistency: one user asks “is this refundable?” and gets “yes, within 30 days”; another user asks the same thing five seconds later and gets “no, all sales final”. Both might be wrong. Both will end up in a support escalation.

The compounding is worse for agents. A planner step at temperature 0.7 picks tool A on Monday and tool B on Tuesday for the same input — the trajectory diverges, the trace looks completely different, and a regression eval that compared average TaskCompletion across the two days flags a “regression” that was just sampling noise. Engineering leaders waste a week chasing a phantom.

The compliance dimension matters too. Healthcare, finance, and legal LLM applications usually require reproducible decisions. An auditor asking “would this model give the same recommendation if rerun?” needs a number, not a vibe. Self-consistency turns that question into a measurable metric.

In 2026-era reasoning models — chain-of-thought, tree-of-thoughts, and self-consistency prompting — self-consistency at evaluation time is also a quality signal. Wei et al.’s self-consistency-prompting paper showed that majority vote across N reasoning chains beats any single chain. Measuring how often the chains agree is the natural extension at eval time.

How FutureAGI Handles the Self-Consistency Evaluation Metric

FutureAGI’s approach is to compute self-consistency in two complementary ways and let engineers pick. Pairwise similarity: sample N responses per input at temperature > 0, then use fi.evals.EmbeddingSimilarity to score every pair and average. A score near 1.0 means the model says the same thing in different words; near 0.5 means it’s flipping. Majority-vote agreement: for tasks with a discrete answer space (yes/no, classification, JSON field values), parse each response, aggregate via fi.evals.AggregatedMetric, and report the modal-answer rate.

For free-form text where neither approach is perfect, FactualConsistency runs an NLI judge across pairs of responses to detect contradictions — stricter than embedding similarity but more meaningful for factual claims. Dataset.add_evaluation() attaches whichever metric you pick to a versioned dataset so you can diff self-consistency across prompt revisions, model versions, and temperature settings.

Concretely: a financial-advice chatbot team running on traceAI-openai samples five responses per question at temperature 0.4, runs EmbeddingSimilarity pairwise, and dashboards self-consistency-by-cohort. When a prompt revision drops the score from 0.91 to 0.74, the team rolls back before user-visible inconsistency complaints arrive. Without self-consistency tracking, the only signal would have been support tickets a week later.

How to Measure or Detect It

Pick the variant that matches your output type:

• Pairwise embedding similarity: fi.evals.EmbeddingSimilarity averaged across all pairs of N samples. Best for free-form text.
• Majority-vote rate: fi.evals.AggregatedMetric on parsed structured outputs; reports the fraction of samples matching the modal answer.
• FactualConsistency: NLI-based contradiction detection across paired responses; surfaces semantic disagreement embedding similarity misses.
• Self-consistency-fail-rate (dashboard signal): the percentage of inputs where pairwise similarity falls below your threshold, sliced by route or model.
• Temperature-sweep curve: self-consistency plotted against decoding temperature — confirms whether instability is a temperature choice or a prompt problem.

Minimal Python:

from fi.evals import EmbeddingSimilarity

sim = EmbeddingSimilarity()
samples = [generate(prompt) for _ in range(5)]
scores = [
    sim.evaluate(input=samples[i], output=samples[j]).score
    for i in range(len(samples))
    for j in range(i + 1, len(samples))
]
self_consistency = sum(scores) / len(scores)

Common Mistakes

• Treating self-consistency as a correctness metric. A model that’s consistently wrong scores 1.0 — pair it with a reference-based metric.
• Sampling at temperature 0. All samples are identical, score is trivially 1.0, signal is zero.
• Using exact-match instead of semantic similarity. Two correct answers can be worded differently; exact-match flags them as inconsistent.
• Ignoring the prompt-paraphrase axis. True robustness means consistency under reworded prompts, not just under repeated identical prompts.
• Sampling N too small. N = 2 has high variance; use N >= 5 for a stable estimate.