What Are Baseline Models?

What Are Baseline Models?

Baseline models are the simple, well-understood reference systems used to anchor model evaluation. Common examples are majority-class predictors, logistic regression, BM25 retrievers, exact-match QA, and a previous LLM checkpoint or prompt. They belong to the model family because they are functioning models, but their role in a pipeline is comparative: any new candidate has to beat the baseline to count as progress. Without baseline models, eval scores are unanchored numbers that drift with luck and dataset choice. FutureAGI evaluates baseline-versus-candidate performance on versioned Dataset runs and gates promotions on per-cohort regression deltas.

Why Baseline Models Matter in Production LLM and Agent Systems

Most LLM teams skip baseline models because they think the modern model is obviously better. That assumption is the source of a long tail of production regressions. A new “smarter” reranker can lose to BM25 on rare-language queries. A 70B fine-tune can lose to the base model on out-of-domain prompts. A multi-agent trajectory can lose to a single-prompt baseline on simple-resolution tickets. None of these are visible without an explicit baseline run.

The pain shows up by role. ML engineers ship prompts that “feel” better and discover months later that user satisfaction trended down. Platform engineers notice the new model is 4x more expensive with no measurable lift. Product managers approve features built on cherry-picked demos. Compliance leads cannot defend a release because there’s no documented baseline to compare to.

In 2026 the case got stronger. Agent stacks add cost and latency on top of the LLM step, so a baseline must include both. A trajectory that scores 5% better than baseline but costs 3x and is 2x slower is a regression in disguise. Multi-agent flows multiply the comparison dimensions, and the only way to avoid winning-on-vibes is to bake baseline runs into the release process.

How FutureAGI Handles Baseline Models

FutureAGI’s approach is to make baseline models a versioned artifact of the platform. A baseline lives as a saved evaluation run — Dataset.add_evaluation results on a specific dataset, evaluator portfolio, and timestamp. Each candidate runs the same evaluator portfolio on the same dataset, and the regression diff is the gate.

A concrete example: a RAG team maintains three baseline models. The first is a “trivial” baseline — answer-from-context with no retrieval — that bounds how good a retriever has to be to justify its cost. The second is a BM25 retrieval baseline. The third is the prior production retrieval pipeline. Every retrieval candidate is scored against all three on the same dataset using Groundedness, ContextRelevance, PrecisionAtK, and NDCG. A candidate must beat all three to ship.

For agent baselines, the FutureAGI workflow uses traceAI to capture trajectory baselines on traceAI-langchain or traceAI-openai-agents. TrajectoryScore, GoalProgress, and StepEfficiency are computed on the prior agent. Candidate agents run via Agent Command Center traffic-mirroring against shadow traffic, scored on the same metrics, and a model fallback keeps the baseline warm during rollout. When the candidate is statistically tied with the baseline, the team adds adversarial cases via the FutureAGI annotation queue and recomputes — strong baselines surface weak candidates fast.

How to Measure or Detect It

The signals are comparative — every candidate row carries a baseline row:

• Per-row score deltas: FactualAccuracy, Groundedness, Faithfulness on candidate minus baseline.
• Per-cohort regression: split deltas by cohort to catch minority-slice regressions.
• PrecisionAtK, NDCG: ranking-quality deltas for retrieval baseline comparisons.
• TrajectoryScore, GoalProgress, StepEfficiency: agent baseline comparisons.
• Latency p99 and cost-per-trace deltas: cost a candidate must beat to justify its quality lift.
• Statistical significance: paired-sample tests on per-row scores to avoid false-positive wins.

Pattern for a per-row baseline-vs-candidate comparison:

from fi.evals import Groundedness

g = Groundedness()
deltas = []
for row in dataset:
    base = g.evaluate(input=row.input, output=row.baseline_output, context=row.context).score
    cand = g.evaluate(input=row.input, output=row.candidate_output, context=row.context).score
    deltas.append(cand - base)

Common Mistakes

• No baseline model at all. “Looks better” is not a release criterion.
• Picking too sophisticated a baseline. A baseline should be the simplest credible alternative; overshooting makes every candidate look bad.
• Letting the baseline dataset drift. If rows change between baseline and candidate runs, the comparison is invalid.
• Reporting only aggregate deltas. Aggregate deltas hide per-cohort regressions, which are exactly the failures users notice.
• Ignoring cost and latency baselines. A quality-equivalent candidate that’s 3x slower is a regression.