What Is Exploratory Data Analysis (EDA)?

What Is Exploratory Data Analysis (EDA)?

Exploratory Data Analysis (EDA) is the open-ended phase of looking at a dataset before modeling: distributions, missing values, outliers, label balance, feature correlations, obvious quality issues. The output is not a model; it is a set of decisions about what to clean, what to drop, and what to log when training runs. For LLM and RAG workflows the surface shifts from numeric histograms to text inspection, retrieval-chunk diversity, eval-score breakdowns, and trace cohort comparison. FutureAGI does not run pandas notebooks; it exposes the eval and trace data the LLM-era EDA needs, queryable through Dataset and traceAI.

Why It Matters in Production LLM and Agent Systems

Skipping EDA is the cheapest way to waste a training run. A team trains a classifier on a leaked feature, runs three weeks of fine-tunes, and discovers test accuracy was an artifact of the leak. A RAG team indexes a corpus that turns out to contain the same document 40 times, blowing up retrieval skew. An agent team builds a planner against a dataset whose user-intent distribution does not match production. The pain falls broadly: ML engineers waste GPU hours; product teams ship models that fail on the first real cohort; finance writes off cloud spend on training runs that never had a chance.

Common production symptoms include: training-test accuracy gaps caused by data leakage; eval-set scores that look strong but production scores that don’t; failure-mode distributions in production that do not appear in any training or eval set; retrievers returning the same chunk for half of all queries because the corpus is dominated by one document.

In 2026-era stacks, EDA is also a continuous activity. Production traces are the new dataset. Teams need to inspect prompt-token-length distributions, retrieved-chunk diversity, model-id usage, eval-fail-rate by cohort, and tool-call frequency before they propose any change. EDA on traces is what tells you which cohort to gather more data for, which prompt to refine, and which guardrail to tighten.

How FutureAGI Handles Exploratory Data Analysis

FutureAGI’s approach is to expose the production and evaluation data so EDA actually surfaces something useful. Dataset is the queryable substrate: import data via file or Hugging Face, add columns and rows, and inspect distributions before a training or eval run. fi.evals evaluators turn unstructured text into queryable scores; aggregate Groundedness, TaskCompletion, AnswerRelevancy, and Toxicity across rows, then slice by cohort or model id. traceAI captures production behavior as OpenTelemetry spans, so EDA on traces is a SQL- or notebook-style operation against gen_ai.request.model, agent.trajectory.step, prompt token counts, and tool latencies. Annotation queues via fi.queues.AnnotationQueue let teams sample low-eval-score rows for human review — a structured form of EDA on production failures.

A practical pattern: a customer-support team is preparing to fine-tune a smaller model. They pull six weeks of production traces into a Dataset, run Groundedness, TaskCompletion, and Toxicity on every row, and dashboard the score distributions per intent category. EDA reveals that 72% of TaskCompletion failures cluster in three intents that together account for 9% of traffic. They route the fine-tune dataset to oversample those intents, set up a regression eval against the canonical golden dataset, and ship through Agent Command Center traffic-mirroring. Unlike training on raw production logs, the EDA-informed dataset corrects an imbalance that would have produced a fluent-but-still-failing model.

How to Measure or Detect It

EDA itself is open-ended, but in a FutureAGI workflow the measurable outputs include:

• Score-distribution histograms: Groundedness, TaskCompletion, AnswerRelevancy distributions per cohort surface skews and outliers.
• Cohort-failure heatmaps (dashboard signal): eval-fail-rate-by-cohort sliced by intent, locale, model id, or route.
• Token-length distributions: prompt and completion token histograms surface long-tail traffic the prompt template was not built for.
• Retrieval-chunk diversity: distinct-chunk-count per query reveals corpus dominance and stale-context issues.
• AggregatedMetric: combines multiple evaluator scores into a single per-row signal, useful for global cohort comparisons.

Minimal Python:

from fi.evals import Groundedness, TaskCompletion

ground = Groundedness()
task = TaskCompletion()
scores = [(row.cohort,
           ground.evaluate(input=row.q, output=row.r, context=row.ctx).score,
           task.evaluate(input=row.q, output=row.r).score) for row in dataset]

Common Mistakes

• Skipping EDA on production traces. Static benchmarks miss the cohorts you actually serve; the production trace is the real dataset.
• One global metric. A 0.78 mean Groundedness hides the 12% cohort scoring 0.31; always slice.
• Looking only at the eval-set. Eval sets curated by engineers do not match real user phrasing or intent distribution.
• Treating EDA as a one-time activity. Production distributions drift; recurring EDA on trace cohorts catches regressions early.
• Ignoring retrieval-chunk diversity. A corpus dominated by one document looks fine on aggregate but produces brittle answers on the long tail.