What Is the Population Stability Index (PSI)?

What Is the Population Stability Index (PSI)?

Population Stability Index (PSI) is a data-drift metric that compares the current distribution of a feature, score, prompt cohort, or dataset slice against a baseline distribution. It bins both populations, calculates the share in each bin, and sums (current - baseline) * ln(current / baseline) across bins. In LLM and agent systems, PSI shows up in dataset monitoring, eval cohorts, and production traces when input mix changes before model quality metrics drop. FutureAGI uses it as an early-warning signal for sdk:Dataset drift.

Why It Matters in Production LLM and Agent Systems

PSI catches a quiet failure mode: the system still works on the old population, while the real traffic has moved. A RAG support bot may keep passing Groundedness on archived eval rows while new users ask refund-policy questions from a region missing in the retrieval corpus. A routing agent may look stable in aggregate while enterprise traffic shifts toward long, tool-heavy workflows that raise cost and timeout risk.

The pain is shared. Developers chase failing prompts without seeing that the input mix changed. SREs notice higher latency p99, retry rate, or token-cost-per-trace, but the dashboard does not explain which cohort moved. Product teams see lower conversion after a channel launch and cannot tell whether the model regressed or the population changed. Compliance teams lose confidence in evaluation evidence if the current dataset no longer matches the approved baseline distribution.

PSI is especially useful for 2026-era multi-step pipelines because drift can enter anywhere: user prompts, retrieved documents, tool outputs, route selections, or judge-model labels. Logs usually show symptoms such as rising eval-fail-rate-by-cohort, more fallback responses, different llm.token_count.prompt distributions, or a spike in manual review for one segment. PSI turns those symptoms into a numeric question: how far did this population move from the reference distribution?

How FutureAGI Handles Population Stability Index (PSI)

FutureAGI’s approach is to treat PSI as a dataset reliability signal, not as a model-quality score. The relevant anchor is sdk:Dataset: the SDK surface fi.datasets.Dataset creates and downloads datasets, adds columns and rows, imports files or Hugging Face data, and attaches evaluations plus eval stats. That makes PSI useful around the dataset boundary where baseline eval sets, current production samples, and regression cohorts meet.

In a FutureAGI workflow, an engineer might keep a January 2026 baseline dataset for a billing-support agent and compare it with this week’s sampled traces. The exact metric is a custom dataset-level stat named population_stability_index, computed for fields such as intent, customer_tier, retriever_top_k_score, prompt_version, and llm.token_count.prompt buckets. The engineer then joins the PSI table with downstream scores from ContextRelevance, Groundedness, or TaskCompletion.

The next action depends on both shift and quality. If PSI on intent is 0.31 but evaluator scores are flat, the team records drift and widens the eval set. If PSI is 0.31 and ContextRelevance drops on the same cohort, the release is blocked, the changed examples are added to a regression eval, and an alert is opened for retriever coverage. Unlike Jensen-Shannon divergence, PSI is easier to explain to operations and risk teams because it shows bucket-level contribution to the final score. We have found that PSI is most useful when it is paired with evaluator deltas, not read as an isolated alarm.

How to Measure or Detect Population Stability Index (PSI)

Measure PSI by comparing bucket shares between a baseline and current population, then segmenting quality metrics by the same buckets.

• Dataset buckets: define stable bins for numeric scores and fixed categories for fields such as intent, region, product tier, and prompt version.
• Dashboard signal: alert on population_stability_index by column, plus eval-fail-rate-by-cohort and token-cost-per-trace for the shifted segment.
• Evaluator pairing: Groundedness evaluates whether the answer stays supported by context; use it to test whether drift changed answer quality.
• Trace field check: compare llm.token_count.prompt, route name, retrieved-document count, or traceAI-langchain span attributes between baseline and current traffic.
• User-feedback proxy: confirm drift against thumbs-down rate, escalation-rate, refund-contact rate, or human-review rate for the same cohort.

Minimal Python:

from fi.evals import Groundedness
import numpy as np

baseline = np.array([0.60, 0.30, 0.10])
current = np.array([0.45, 0.40, 0.15])
psi = ((current - baseline) * np.log(current / baseline)).sum()
paired_eval = Groundedness()
print({"population_stability_index": float(psi), "paired_eval": paired_eval.__class__.__name__})

Common Mistakes

• Using unstable bins. If bin edges change every run, PSI measures binning noise instead of population shift.
• Ignoring sample size. A high PSI on 40 rows is an investigation trigger, not a release blocker.
• Treating 0.1 and 0.25 as universal laws. Calibrate thresholds by feature importance, seasonality, and evaluator impact.
• Monitoring only model inputs. Agent drift also appears in retrieved context, tool outputs, route selections, and judge labels.
• Resetting the baseline too often. Moving the reference distribution after every release hides gradual drift and weakens regression evidence.