What Is Eval-Driven Development?

What Is Eval-Driven Development?

Eval-driven development (EDD) is the practice of writing the evaluation suite before — or alongside — the LLM feature it grades, and treating the suite as the release spec. The workflow: enumerate failure modes, write one evaluator per mode, build a small golden dataset that exercises each, and require a green run before shipping. EDD is the LLM-system analogue of test-driven development. The eval suite plays the role of the test suite; the golden dataset plays the role of test fixtures; the release gate is whether AggregatedMetric clears its threshold. Without EDD, prompt changes ship on opinion.

Why Eval-Driven Development Matters in Production

The thing EDD fixes is the prompt-engineering chaos most teams live in. Without it, a typical release looks like: someone tweaks the system prompt, the demo looks good, the change ships. Two days later support tickets spike. Nobody can reproduce the regression because there is no canonical test set. The fix is another prompt tweak, the demo looks good, and the cycle continues.

The pain hits four roles. ML engineers waste time on regression archaeology — bisecting prompt edits manually. Product managers can’t answer “is this version actually better?” without running another demo. SREs see cost and latency drift quarter-over-quarter with no quality countermeasure. Compliance is asked for evidence the model behaves consistently and has none.

EDD inverts the loop. Before the prompt change, an engineer adds a new test row to the golden dataset capturing the desired behavior, plus the relevant evaluator (often Groundedness or TaskCompletion or a CustomEvaluation). Then they edit the prompt. The eval suite tells them whether the change broke anything. Multiply by 50 features and you have a system that doesn’t regress silently.

For 2026 agent stacks, EDD is the only way to keep multi-step systems sane. Each step (planner, retriever, tool selection, response composition) gets its own evaluator. A change to the retriever that improves answer quality but breaks tool-call accuracy gets caught — without EDD, this is the kind of regression that surfaces three weeks later as “agents take 4× longer.” Comparable open-source workflows around DSPy and LangSmith approach this from different angles; FutureAGI’s approach is to make Dataset.add_evaluation() the unified surface so the same evaluators run pre-release and live.

How FutureAGI Handles Eval-Driven Development

FutureAGI’s approach is to ship the EDD primitives as one coherent SDK. fi.datasets.Dataset is the test fixture container — create it once, add columns and rows over time, version it, import from CSV/Hugging Face. Dataset.add_evaluation() attaches one or more fi.evals evaluators to the dataset, runs them across all rows in parallel, and stores the results columnwise. AggregatedMetric defines the release gate. CI calls the SDK; a failing aggregate fails the build.

For ongoing EDD, traceAI integrations (traceAI-langchain, traceAI-openai-agents, traceAI-llamaindex) sample production into the same dataset, so the corpus grows organically with real failure cases users hit. The fi.queues.AnnotationQueue lets humans label the new cases, which then get promoted into the golden dataset.

A real flow: a team building a SQL-generation agent declares three failure modes — wrong table joined, hallucinated column, query syntax invalid. They write three evaluators (CustomEvaluation for table correctness, CustomEvaluation for column existence against schema, TextToSQL built-in for syntax). They build a 200-row golden dataset over two days. CI runs the suite on every PR; passing requires aggregate ≥0.85. Eight weeks in, the dataset has grown to 2,400 rows from production samples, and the team is shipping prompt and model changes weekly with confidence.

How to Measure or Detect EDD Adoption

EDD itself produces measurable signals:

• Coverage: % of known failure modes that have a corresponding evaluator. Target 100% for production.
• Golden-dataset size and growth rate: dataset size over time; healthy EDD adds rows weekly from production samples.
• AggregatedMetric pass rate on PRs: % of PRs that hit the gate first try. Sub-50% means the gate is too tight or the suite is flaky.
• Time-to-detect-regression: hours between a regression landing in main and the eval flagging it. EDD shrinks this from weeks to PR-time.
• Eval coverage of production traces: % of live traces that match a golden-dataset row pattern. Low coverage means the dataset is stale.

Minimal Python:

from fi.datasets import Dataset
from fi.evals import Groundedness, JSONValidation, AggregatedMetric

ds = Dataset.get("sql-agent-golden")
ds.add_evaluation(
    AggregatedMetric([Groundedness(), JSONValidation()], weights=[0.7, 0.3]),
    threshold=0.85,
)

Common Mistakes

• Writing evals after shipping. This is regression archaeology, not EDD. Evals must precede or accompany the feature.
• Tiny golden dataset that doesn’t exercise edge cases. 20 rows of happy-path is not a spec.
• Treating the eval suite as advisory. If a failing aggregate doesn’t block deploys, EDD has no teeth.
• Skipping production-trace promotion. A static golden dataset goes stale; promote real failures into it weekly.
• No threshold per evaluator. Aggregate-only gating hides single-metric regressions.