What Is the Kolmogorov-Smirnov Test?

What Is the Kolmogorov-Smirnov Test?

The Kolmogorov-Smirnov test is a nonparametric statistical test for comparing one numeric sample with a reference distribution, or two samples with each other. It belongs to the data family because it detects distribution drift before that drift corrupts evals, monitoring, or release decisions. In FutureAGI workflows, it usually appears around datasets and production traces, where engineers compare current prompt lengths, embedding scores, latencies, evaluator scores, or cohort metrics against an approved baseline.

Why the Kolmogorov-Smirnov Test Matters in Production LLM and Agent Systems

Distribution drift turns old reliability evidence into a weak release gate. A support agent can pass last month’s eval set while current traffic contains longer prompts, new account tiers, or retriever scores from a changed index. A routing system can look healthy on aggregate while one cohort receives low-confidence answers. The failure mode is silent data drift: the system does not crash, but thresholds, eval pass rates, and dashboards no longer describe the traffic being served.

Engineers feel this as confusing regression signals. Developers see a prompt patch fail without an obvious code reason. SREs see p99 latency, token cost per trace, or escalation rate move before any model error appears. Product teams see answer quality degrade in one tenant or locale while the global pass rate stays flat. Compliance teams lose confidence that audited cases still represent production behavior.

The KS test is especially useful in 2026-era agent pipelines because many steps produce numeric traces: planner step count, retrieval score, tool latency, token count, refusal score, evaluator score, and final response length. A shift in any one of those distributions can change downstream behavior. For example, a longer prompt distribution can push context over budget, which lowers retrieval quality and raises hallucination risk. The KS statistic gives teams an early, cohort-aware signal that the traffic distribution changed before the final answer metric collapses.

How FutureAGI Uses the Kolmogorov-Smirnov Test

FutureAGI does not expose the Kolmogorov-Smirnov test as a named evaluator; the anchor for this entry is none. FutureAGI’s approach is to use it as an external data diagnostic around fi.datasets.Dataset and trace evidence, then connect the result to concrete eval and monitoring actions. The test answers a narrow question: “did this numeric distribution move?” FutureAGI then helps answer the operational question: “does that movement matter for model, retriever, or agent quality?”

A practical workflow starts with a support RAG agent instrumented through traceAI-langchain. The team stores an approved reference window in a dataset and compares it with current production samples for llm.token_count.prompt, retrieval similarity score, response length, latency, and eval score by cohort. They run a two-sample KS test for each field and alert when the statistic crosses the team’s threshold and the p-value is below the agreed cutoff.

What happens next is not “change the model.” The engineer opens the affected cohort, inspects traces, and reruns adjacent FutureAGI evaluators such as ContextRelevance and Groundedness on the failing rows. If prompt-token distribution shifted because a retriever started returning longer policy chunks, the fix is likely a retrieval or chunking change. If evaluator-score distribution shifted without input drift, the prompt or provider changed. Unlike population stability index, which depends on binning choices, KS can catch a sharp local shift in the cumulative distribution without choosing buckets first.

How to Measure or Detect the Kolmogorov-Smirnov Test

Measure KS only on ordered numeric values. For LLM systems, good inputs include token counts, latencies, embedding distances, retrieval scores, evaluator scores, confidence scores, response lengths, and per-step agent counts.

• Reference window: define the baseline distribution from a reviewed dataset, release candidate, or stable production period.
• Current window: sample the same field from current traffic, segmented by tenant, locale, route, model, prompt version, and cohort.
• KS statistic: report the maximum cumulative-distribution gap; larger values mean stronger observed distribution movement.
• P-value: use it as uncertainty evidence, not effect size. Large samples can make tiny shifts statistically significant.
• FutureAGI pairing: compare KS alerts with eval-fail-rate-by-cohort, p99 latency, token-cost-per-trace, thumbs-down rate, and escalation rate.

from scipy.stats import ks_2samp

stat, pvalue = ks_2samp(reference_scores, current_scores)
if stat > 0.12 and pvalue < 0.01:
    alert("distribution shift", stat=stat, pvalue=pvalue)

The test tells you that two distributions differ; it does not explain why. Pair it with quantile plots, cohort slices, trace examples, and evaluator results before opening a release blocker.

Common Mistakes

• Using KS on raw text or categories. It needs ordered numeric values; convert text to measurable scores, lengths, distances, or evaluator outputs first.
• Treating p-value as business impact. With enough traffic, tiny harmless shifts look significant. Always report the KS statistic and practical threshold.
• Checking aggregate traffic only. Agent drift often hides inside one route, tool path, account tier, locale, or prompt version.
• Ignoring ties and discrete scores. Heavily rounded confidence scores may need permutation checks or a complementary metric.
• Acting on KS alone. Pair the alert with traces, cohort evals, and user-feedback movement before changing prompts or models.