What Is SafetyBench?

What Is SafetyBench?

SafetyBench is a bilingual LLM safety benchmark that tests safety understanding with 11,435 multiple-choice questions across seven risk categories. It belongs to the AI security evaluation family and shows up in eval pipelines before a model, chatbot, or agent route ships. In FutureAGI, teams convert SafetyBench-style cases into category-level regression evals, attach evaluator scores such as ContentSafety and BiasDetection, and inspect failures in production traces or guardrail reviews.

Why SafetyBench matters in production LLM and agent systems

SafetyBench matters because a model can pass general QA tests while still mishandling safety decisions. The concrete failure is not a syntax error or a 500 response. It is a fluent answer that chooses an unsafe action, misses bias, normalizes harmful advice, leaks privacy context, or misunderstands legal and ethical boundaries.

Developers feel the pain when a model upgrade improves reasoning benchmarks but regresses on a specific safety category. SREs see normal latency and success-rate graphs, yet abuse reports, escalations, blocked sessions, and review queues rise. Compliance and security teams need evidence that each risk category was tested, not a single average score. Product teams need to know whether the model is over-refusing harmless requests or under-refusing risky ones.

The issue is sharper for agentic systems. A chatbot that misclassifies a privacy question may only write a bad answer. An agent can call tools, send messages, update records, browse third-party content, or carry the mistaken judgment into later steps. In 2026 multi-step pipelines, SafetyBench-style coverage should be treated as a category map: where does the route fail, in which language, under which prompt version, and after which retrieval or tool step? Useful symptoms include category fail rate, refusal inconsistency, unsafe-action rate, guardrail false negatives, and a spike in human review for health, legal, bias, or privacy cases.

How FutureAGI handles SafetyBench

FutureAGI handles SafetyBench as an eval-dataset workflow anchored to eval:*, not as a standalone leaderboard number. An engineer imports SafetyBench-style cases into a FutureAGI dataset with columns such as question, options, expected_choice, risk_category, language, policy_scope, model_route, and prompt_version. The run stores model_choice, category accuracy, eval-fail-rate-by-cohort, trace_id, and any guardrail action attached to the same route.

The practical evaluator mapping is category-specific. Offensiveness and harmful-content cases map to ContentSafety and IsHarmfulAdvice. Unfairness and bias cases map to BiasDetection. Privacy and property cases can be reviewed with DataPrivacyCompliance, plus PII-oriented checks when the case includes sensitive data. The point is not to claim that one evaluator is “SafetyBench.” The point is to preserve the benchmark category, then attach FutureAGI evaluator evidence to the exact model and prompt path that will run in production.

A real workflow: a LangChain support agent routes through Agent Command Center with a pre-guardrail for risky input and a post-guardrail for unsafe answers. The langchain traceAI integration captures the model call, tool span, prompt version, and trace id. If the legal or health category drops below threshold, the engineer blocks the model route, inspects failed examples, adjusts the refusal prompt, adds a guardrail rule, or creates a regression eval for the failed cohort.

FutureAGI’s approach is to treat SafetyBench as coverage analysis, not release permission. Unlike HarmBench, which stresses harmful-request compliance and jailbreak behavior, SafetyBench is best at finding broad safety-understanding gaps before deeper red-team and agent-action tests.

How to measure or detect SafetyBench gaps

Measure SafetyBench at category level before calculating any aggregate score:

• Benchmark accuracy — percentage of questions where model_choice matches expected_choice, sliced by risk_category, language, model, route, and prompt version.
• Evaluator overlay — ContentSafety, BiasDetection, IsHarmfulAdvice, and DataPrivacyCompliance add semantic signals to failed or high-risk cases.
• Dashboard signal — track eval-fail-rate-by-cohort, guardrail block rate, post-guardrail warn rate, and release-blocking category regressions.
• Trace evidence — attach trace_id, tool name, retrieval source, route, and guardrail action so each failure becomes a debuggable production path.
• User-feedback proxy — compare benchmark gaps with thumbs-down rate, escalation rate, safety review backlog, and appealed guardrail decisions.

from fi.evals import ContentSafety, BiasDetection, IsHarmfulAdvice

response = "The safest answer is option C because ..."
scores = {
    "content_safety": ContentSafety().evaluate(input=response).score,
    "bias": BiasDetection().evaluate(input=response).score,
    "harmful_advice": IsHarmfulAdvice().evaluate(input=response).score,
}
print(scores)

Use at least one release gate that cannot be hidden by the average, such as zero critical failures in privacy/property cases and no category dropping more than two percentage points from the approved baseline.

Common mistakes

Teams usually misuse SafetyBench when they turn a broad benchmark into a blanket safety claim. The fixes are operational: preserve categories, keep traces, and run adjacent tests when the production system can act.

• Treating multiple-choice accuracy as refusal quality; a model can identify unsafe options and still generate unsafe advice in chat.
• Averaging across the seven categories; privacy/property failures can disappear behind strong ethics or offensiveness scores.
• Testing English only; SafetyBench is bilingual, and category errors often move with translation, locale, and policy wording.
• Using SafetyBench as a prompt-injection test; direct and indirect injection need PromptInjection, ProtectFlash, and route-level probes.
• Shipping without trace links; a failed category score must point to model route, prompt version, and guardrail decision.