What Is a Pipeline?

What Is a Pipeline?

A machine-learning pipeline is the chained sequence of steps that turns raw input into a model output and a stored artifact. A training pipeline runs ingestion, preprocessing, feature engineering, training, evaluation, model registration, and deployment. An inference pipeline runs request handling, prompt assembly, retrieval, model invocation, tool calls, post-processing, and guardrails. Each step is a measurable surface with its own latency, cost, and quality. FutureAGI plugs into both shapes: traceAI for step-level spans, fi.evals for boundary evaluators, and fi.datasets.Dataset for reproducible runs.

Why Pipelines Matter in Production LLM and Agent Systems

A pipeline is where most production failures actually live. The model is fine; the chunker silently truncates documents, the reranker drops the relevant passage, the tool times out, the post-processor fails to coerce JSON, or the guardrail blocks a benign response. None of these are “model bugs” — they are pipeline bugs, and they account for the majority of incidents in real LLM systems.

The pain is shared across roles. ML engineers see “the model regressed” claims that are actually retrieval drift after a corpus update. SREs chase a latency spike that turns out to be a slow tool call upstream of the model. Product managers see thumbs-down rates rise on what looks like the same model and prompt — the change was a chunking parameter. Compliance teams need to prove the data path for a regulated decision; without per-step trace evidence, they cannot.

In 2026 agent stacks the pipeline becomes a graph, not a line. A single user request can fan out into a planner, a retriever, three tool calls, a critique, and a final response. Each step is itself a pipeline. Eval-fail-rate-by-cohort, p99 latency, token cost, and guardrail decisions need to be reported per step, per route, per model variant, or you cannot find the broken link.

How FutureAGI Instruments and Evaluates Pipelines

FutureAGI’s approach is to make every pipeline step a measurable boundary. The named anchors are traceAI integrations (traceAI-langchain, traceAI-llamaindex, traceAI-vllm, traceAI-mcp, traceAI-livekit, etc.) for step-level OTel spans, plus fi.evals evaluators applied at each boundary. fi.datasets.Dataset stores reproducible eval runs with Dataset.add_evaluation. Agent Command Center adds runtime primitives such as pre-guardrail, post-guardrail, model fallback, semantic-cache, and traffic-mirroring that themselves become pipeline steps.

Real example: a RAG team’s pipeline runs retrieval → reranker → prompt assembly → model call → JSON post-processor. They instrument each step with traceAI-langchain. After the retrieval step they attach ContextRelevance. After the model call they attach Groundedness and AnswerRelevancy. After the post-processor they attach JSONValidation. FutureAGI surfaces a per-step scorecard: retrieval is fine, the reranker is dropping relevant chunks 8% of the time, the model is grounded when given good context, and JSON is valid 99.7%. The fix is at the reranker, not the model. Compared with one end-to-end accuracy number, this points to the broken link.

How to Measure or Detect It

Measure pipelines per step, per route, per release.

• Per-step latency — p50 and p99 for each span; outliers usually reveal a misconfigured step.
• Per-step token usage — llm.token_count.prompt and llm.token_count.completion aggregated by step type.
• Per-step evaluator score — ContextRelevance after retrieval, Groundedness/AnswerRelevancy after generation, JSONValidation after post-processing.
• Step success rate — fraction of spans that complete without error, blocked by guardrail, or hit fallback.
• Trace fields — agent.trajectory.step, route name, model.version, prompt.version, tool name, retrieval.score.
• Eval-fail-rate-by-step — group eval failures by step type to see where the pipeline breaks.

from fi.evals import Groundedness, JSONValidation

g = Groundedness().evaluate(response=resp, context=ctx)
j = JSONValidation().evaluate(output=resp, schema=schema)
print(g.score, j.score)

Common Mistakes

• Reporting only end-to-end metrics. Without per-step scores, regressions become guesswork.
• Skipping retrieval evaluation. Retrieval bugs are the most common LLM-pipeline failure mode and the easiest to miss.
• Letting steps share configuration silently. A prompt-version change in step 3 should be visible to all steps that depend on it.
• No reproducibility on dataset versions. Eval results from two months ago are useless if the dataset version was overwritten.
• Treating training and inference pipelines the same. They have different reliability profiles and need different SLOs.