What Is SHAP Values?

What Is SHAP Values?

SHAP values are per-feature contribution scores for a single model prediction, produced by the SHAP (SHapley Additive exPlanations) framework. Each value tells you how much that feature pushed the prediction above or below the model’s baseline output, with one strict guarantee: the values sum exactly to the difference between the prediction and the baseline. Mathematically they are Shapley values from cooperative game theory adapted to ML attribution. SHAP values are widely used on tree-based and tabular models for debugging, fairness audits, and per-instance explanations to regulators and end-users.

Why It Matters in Production LLM and Agent Systems

Raw model scores are illegible to most stakeholders. SHAP values translate “the model returned 0.74” into “feature A added 0.18, feature B subtracted 0.07, baseline was 0.63.” That decomposition is what auditors, support teams, and disputing customers actually need.

The pain of not having per-prediction attribution shows up across roles. A compliance lead under the EU AI Act has to explain why a high-risk decision rejected an applicant — global feature importance is not enough; she needs this applicant’s number. A platform engineer chases a “model degraded” alert and cannot tell which feature’s distribution shifted because she has only aggregate scores. A product team faces customer escalations they cannot defend without per-decision rationale.

In 2026 agent stacks, SHAP values matter most where agents delegate to tabular or tree-based scorers — credit-risk routing, fraud triage, support-ticket prioritization. The agent’s own LLM reasoning is opaque, but the scorer it calls into can ship SHAP values per call. A well-instrumented agent logs the scorer’s SHAP attributions alongside the agent’s tool-call output, so when a downstream evaluation flags a biased outcome, the team can trace the bias to a specific feature on a specific call — not just to a black box.

How FutureAGI Handles SHAP Values

FutureAGI does not compute SHAP values; that lives in the modeling library. We make them queryable, auditable, and testable downstream. At the trace level, you log SHAP attributions as span metadata via fi.client.Client.log, so every prediction’s SHAP vector is searchable in production. At the dataset level, Dataset.add_evaluation lets you attach a CustomEvaluation that runs over the SHAP values — for example, flag any prediction where the absolute SHAP value of a protected attribute exceeds 0.05, or where the top-three SHAP features change between model versions on identical inputs. At the audit level, the FutureAGI audit-log preserves inputs, outputs, and SHAP metadata together, so months later a regulator request can be answered with the exact attribution computed at the time of decision.

Concretely: a fintech team trains an XGBoost credit-risk model, computes SHAP values on every prediction, and serves the model behind a traceAI integration that records shap.values as a span attribute. They run a nightly CustomEvaluation against the prior week’s predictions: if the mean absolute SHAP value of zip_code (a known proxy-bias feature) crosses 0.04, the eval fails and a regression-eval is triggered. FutureAGI’s BiasDetection evaluator then scores outcome disparity across geographic cohorts to confirm whether the SHAP signal corresponds to real disparate impact. The attribution and the outcome eval together produce an audit trail.

How to Measure or Detect It

SHAP-derived signals to wire into reliability dashboards:

• Per-feature SHAP magnitude — the absolute SHAP value for one feature on one prediction; spike-detection on protected features.
• Top-K feature stability — the set of top-3 SHAP features per prediction across model versions; instability signals a behavior change.
• Cohort-mean SHAP — average SHAP value of a feature across a user cohort; surfaces systematic differences in how the model treats groups.
• BiasDetection — confirms whether SHAP-attributed differences correspond to disparate outcome rates across cohorts.
• SHAP coverage — percentage of predictions logged with SHAP attached; below 100% means the audit trail has gaps.

Minimal Python — log SHAP alongside the prediction:

import shap
from fi.client import Client

explainer = shap.TreeExplainer(model)
sv = explainer.shap_values(X_row)
Client().log(
    inputs=X_row.to_dict(),
    outputs={"score": float(prediction)},
    metadata={"shap_values": sv.tolist(), "baseline": float(explainer.expected_value)},
)

Common Mistakes

• Mixing SHAP value with prediction direction. A negative SHAP value reduces the score; readers often assume “negative = bad.” Always show the baseline.
• Computing on a sample, applying globally. SHAP values for a small sample do not generalize; per-prediction attribution is per-prediction.
• Skipping correlation analysis. Two correlated features split SHAP credit unintuitively; a “clean” zip-code attribution can still hide proxy bias.
• Logging only the top-1 feature. Audits need the full vector. Truncating to top-N hides the long tail of small contributors that may matter for some users.
• No reproducibility test. SHAP values for the same input on the same model version should be identical. Drift on identical inputs is a silent model swap.