90% less manual effort in meeting summarization evaluation

The Challenge

Various AI models can generate meeting summaries, but selecting the best-performing model requires systematic evaluation. Organizations faced three key obstacles:

• Subjective evaluation - Human biases led to inconsistent results when comparing AI-generated summaries
• Scalability issues - Multi-model assessment was time-consuming, making it infeasible to evaluate at scale
• No systematic metrics - Absence of standardized quality metrics made comparison unreliable

The Solution

Future AGI’s evaluation framework enabled objective, automated assessment across multiple dimensions:

Multi-Model Summary Generation

Meeting transcripts were processed through multiple AI models simultaneously, generating competing summaries for side-by-side evaluation.

Evaluation Criteria

Four core metrics were defined using the evaluation SDK:

1. Relevance - How well the summary captures the key points from the meeting
2. Coherence - Logical flow and readability of the generated summary
3. Brevity - Conciseness without losing critical information
4. Coverage - Completeness of topic coverage from the original transcript

Deterministic Evaluation

Using the MLLMTestCase framework, deterministic evaluation modules scored each model’s output against the defined criteria, aggregating results for objective comparison.

Dashboard & Model Comparison

Scores were aggregated and visualized in a unified dashboard, making it simple to identify top performers across all metrics.

The Results

• 90% reduction in manual evaluation effort
• Objective scoring eliminating human bias from model selection
• 20% improvement in internal team efficiency
• 10x increase in summary evaluation capacity
• Data-driven model selection replacing subjective judgment with measurable criteria