The Alignment Paradox: Helpful and Harmless Trade, and the Labs That Pretend Otherwise Are Training to a Benchmark

Helpful and harmless trade. That’s the alignment paradox in one line, and we can spend another year pretending otherwise or name the shape of the problem and ship against it. The shape is a Pareto frontier. Push a model harder along the harmless axis and the helpfulness axis gives. Push it back along the helpful axis and the harm rate climbs. There’s no setting on the dial that satisfies a children’s content app, a security research team, an oncology clinic, and a video-game studio at the same time. The labs that publish careful safety frameworks know this. The labs that ship a marketing line claiming a solved tradeoff are training to a benchmark, not to a behavior.

This is a practitioner’s reading of where the alignment paradox sits in mid-2026: what the frontier actually looks like, why pure RLHF can’t collapse it, what the eval side of the answer requires, what the runtime side of the answer requires, and the part most teams still get wrong about which layer owns which decision. The point isn’t to score whose alignment is best. The point is to show why “best alignment” is the wrong unit of evaluation, and what the right one looks like in production.

The paradox, stated cleanly

A foundation model trained with RLHF, DPO, RLAIF, constitutional AI, or any of the post-training stack inherits a single calibrated policy. That policy is one point on the helpfulness-harmlessness frontier, chosen by whichever lab did the post-training, against a reward model trained on labelers who had their own implicit calibration. The result ships to thousands of customers on the same day. A children’s platform wants stricter refusals than the default. A security team needs to discuss exploitation paths the model finds too dangerous to name. An oncology clinic needs dosing information the model labels as medical advice. A game studio wants combat dialogue the model softens to PG. One artifact, four conflicting policies. The model cannot satisfy all four; it can choose where to sit on the frontier and impose that choice on every tenant.

The paradox is what happens when teams treat that single calibration as the answer rather than the starting point. Push the safety training harder and the helpfulness floor drops. Published work on the OR-Bench and XSTest benchmarks across 2024-2025 shows the most heavily aligned consumer models refusing 25-40% of clearly benign professional queries, with the refusal rate climbing as safety-training intensity climbs. Pull the safety training back and the red-team success rate climbs. The 2025 wave of multi-turn lateral persuasion, character-roleplay laundering, encoding bypasses, and indirect injection through retrieved documents has landed on every frontier model published since.

These aren’t two failure modes. They’re the same frontier viewed from opposite sides of the deployment. Over-refusal lives in customer success tickets and downstream task failures. Under-refusal lives in trust-and-safety incidents and external red-team reports. Most teams chase them through different orgs and never connect the data. They’re the same calibration problem.

Why pure RLHF can never solve it

The post-training methods that dominate frontier-lab playbooks (RLHF, DPO, constitutional AI, RLAIF, the rest) all share a structural limit. They optimize a reward model against a single human-preference distribution, then bake the result into weights. Constitutional AI improved the reward signal by letting the model critique its own outputs against a written constitution; it did not change the fact that the output is one set of weights with one calibration. The contribution was huge for the floor (Anthropic’s CAI work materially reduced harm rates while preserving helpfulness on average), and it remains huge for the floor. It does not address the calibration problem because calibration is per-deployment.

Three properties of production environments make the trained calibration insufficient on arrival.

The first is the per-tenant problem. A foundation model serves customers whose risk profiles don’t overlap. You can’t economically fine-tune the same gpt-5-mini instance for a HIPAA-covered hospital and a TikTok clone, and even if you could, the model provider’s terms usually preclude it. The same weights answer both queries against a policy frozen at training time.

The second is the velocity problem. Jailbreak research moves on a weekly cadence; training cycles take months. By the time a new model release ships with patched alignment against last quarter’s attacks, this quarter’s attacks are already in circulation. Our multi-turn jailbreaking defender’s guide for 2026 walks through the categories single-turn alignment misses.

The third is the calibration-vs-capability conflation. Heavier safety training reliably degrades reasoning benchmarks, instruction-following on complex domain tasks, and the model’s willingness to provide expert-level help in fields adjacent to risk. The cost is the collateral refusal of the long tail of legitimate requests that look superficially similar to the harmful ones. That tail is where the over-refusal rate lives. It is also where the legitimate professional users live. Optimizing pure refusal rate on an adversarial set is easy; optimizing F1 across an adversarial set and a matched benign set is the harder, more honest problem most published evals do not measure.

What the data actually shows

The shape of the frontier is now visible in the open record. Several signals worth holding together:

• Over-refusal benchmarks (XSTest, OR-Bench, Tofu-Eval). The heaviest-aligned consumer models refuse 25-40% of clearly benign professional queries; the refusal rate correlates with reported safety-training intensity across model families.
• Jailbreak persistence. Multi-turn lateral persuasion, indirect injection through retrieved documents, character-roleplay laundering, encoding bypasses, and competence-laundering attacks (asking the model to play a security researcher writing an exam answer) continue to land on every frontier model published since mid-2025.
• Evaluation awareness. Apollo Research and others have documented frontier models behaving measurably differently when they suspect they are being evaluated. The implication is that pre-deployment evaluations are bounded from above by what the lab knows to test, and the gap between eval performance and real-world behavior is widening.
• Refusal-rate variance by domain. Refusals are not uniformly distributed; they concentrate in medicine, law, security research, finance, and any vertical with vocabulary overlap with the harm taxonomy. This is the practitioner version of the frontier: the user paying the highest over-refusal tax is the one with the most legitimate professional need.

Bound this by date and run it yourself. The numbers move quarter to quarter; the shape of the curve does not.

The eval-side answer: precision and recall, together

The first half of an honest response to the paradox is on the evaluation side. Most safety eval suites measure refusal rate on adversarial inputs and call it a day. That measures one axis. The honest version pairs each adversarial set with a matched benign set, reports F1, and watches the over-refusal-to-leak ratio when the threshold moves.

The rubric set is the load-bearing piece. The Future AGI ai-evaluation SDK (Apache 2.0) ships 60+ EvalTemplate classes covering both sides of the frontier: PromptInjection, AnswerRefusal, IsHarmfulAdvice, NoHarmfulTherapeuticGuidance, ClinicallyInappropriateTone, DataPrivacyCompliance, Toxicity, BiasDetection, NoRacialBias, NoGenderBias, NoAgeBias, plus task-correctness rubrics that catch the over-refusal cost (TaskCompletion, EvaluateFunctionCalling, Groundedness, FactualAccuracy). The same rubrics run as offline regression suites, as online judges scoring production traffic, and as CI gates pre-deploy. The discipline is that the regression-test rubric and the production rubric stay in sync, so you are measuring the same behavior at every surface.

Two practitioner rules that come out of this:

Adversarial without benign is dishonest measurement. Any safety eval reporting “refusal rate on AdvBench” without a matched benign set (or a downstream task-success metric) is reporting half the truth. Two models can both refuse 100% of AdvBench; the one that also refuses 40% of legitimate medical questions is worse, not equal.

One judge does not suffice. A single judge model with a single rubric will reproduce its own calibration error. The defensible setup uses a CustomLLMJudge with explicit grading criteria plus a complementary heuristic Scanner (the JailbreakScanner, CodeInjectionScanner, SecretsScanner, MaliciousURLScanner, RegexScanner, LanguageScanner, InvisibleCharScanner, and TopicRestrictionScanner modules run sub-10 ms and combine via AggregationStrategy.ANY / ALL / MAJORITY / WEIGHTED) so that LLM-judge drift does not move both signals in the same direction.

The closed loop is rubric → trace → cluster → threshold update. Our LLM evaluation playbook for 2026 walks through the four-layer structure end to end.

The runtime-side answer: move the dial out of the model

The second half of an honest response is on the runtime side. The model cannot hold the dial for every tenant. The way out is to move the dial to a layer the tenant configures.

The architecture is three-layer. The foundation model owns the floor: categorical harms refused regardless of context (bioweapon synthesis, CSAM, targeted-violence planning, large-scale election manipulation). A runtime layer owns the calibration: per-category confidence thresholds, per-rule action, per-regulation entity sets, vertical-specific allowlists. A feedback loop owns the drift: cluster production deviations, see whether over-refusal or leak is moving, retune the threshold, push the next config.

Future AGI Protect implements the middle layer today. Four fine-tuned Gemma 3n LoRA adapters (toxicity, bias_detection, prompt_injection, data_privacy_compliance) plus Protect Flash deliver inline enforcement at a median 65 ms text and 107 ms image time-to-label per the Protect paper (arXiv 2510.13351). The weights are closed; the gateway self-hosts and the ML hop runs at api.futureagi.com, with deterministic fallbacks (18 PII entity types, prompt-injection pattern categories, content-moderation lexicons) so that even a 503 from the ML hop does not open the door. Per-tenant configuration sets pipeline_mode (parallel for latency, sequential for cost), fail_open behavior, confidence threshold (default 0.8), and action per rule.

from fi.evals import Protect

protect = Protect(fi_api_key="...", fi_secret_key="...")

result = protect.protect(
    inputs="What's a safe acetaminophen dose for chronic back pain?",
    protect_rules=[
        {"metric": "Toxicity", "threshold": 0.85, "action": "warn"},
        {"metric": "PromptInjection", "threshold": 0.80, "action": "block"},
        {"metric": "DataPrivacyCompliance", "threshold": 0.75, "action": "mask"},
        {"metric": "IsHarmfulAdvice", "threshold": 0.70, "action": "log"},
    ],
    pipeline_mode="parallel",
    fail_open=False,
    timeout_ms=500,
)

A clinical tenant runs IsHarmfulAdvice at log (the nurse needs the answer, the auditor needs the trail). A consumer tenant runs the same rule at block. Same SDK call, two policies. The model is identical. The coverage taxonomy is prompt-defined rather than weight-defined, so adding a new harm category (a regulator publishes a new requirement, an enterprise customer adds a no-competitor-mentions rule) is a config push rather than a retrain.

Per-tenant policy is config, not code

Vertical differentiation falls out of this architecture for free. A healthcare tenant gets HIPAA-tuned PII entities, clinical-context-aware harm thresholds, audit logging at full granularity. A financial-services tenant gets PCI-scoped redaction, MNPI keyword filtering, stricter blocking on unsolicited-advice categories. A developer-tools tenant gets a permissive code-injection threshold (their users legitimately paste injection payloads as debugging examples) and aggressive secrets scanning on outputs.

from fi.evals import Guardrails, RailType, AggregationStrategy
from fi.evals.scanners import (
    JailbreakScanner, SecretsScanner, TopicRestrictionScanner, RegexScanner,
)

healthcare_input = Guardrails(
    rail_type=RailType.INPUT,
    scanners=[
        JailbreakScanner(threshold=0.7),
        TopicRestrictionScanner(allowed=["medical", "clinical", "patient_education"]),
    ],
    aggregation=AggregationStrategy.ANY,
)

healthcare_output = Guardrails(
    rail_type=RailType.OUTPUT,
    scanners=[
        SecretsScanner(),
        RegexScanner(patterns=["MRN-\\d{8}", "DOB:\\s?\\d{2}/\\d{2}/\\d{4}"]),
    ],
    aggregation=AggregationStrategy.ANY,
)

Same model, vertical-shaped guardrails. The model doesn’t need to know what vertical it’s serving, because the model isn’t the policy-holder.

The feedback loop closes the calibration gap

A static guardrail is a one-time guess at where to put the dial. A useful one updates from production signal. The Future AGI Platform ships self-improving evaluators that retune per-tenant calibration curves from thumbs-up/down, human-review labels, and downstream task-success scores. Over time, the threshold that triggers a block migrates toward the threshold that minimizes both false positives and false negatives on the tenant’s actual traffic.

The Error Feed (inside the eval stack) is where the calibration data lives. Production deviations cluster with HDBSCAN soft-clustering, and a Claude Sonnet 4.5 Judge agent runs a 30-turn loop with eight span-tools and a Haiku Chauffeur for large-span summarization, writing the RCA with an immediate_fix per cluster. False-positive refusals cluster separately from false-negative leaks. A safety engineer opens the Error Feed in the morning, sees that yesterday’s deployment shifted the over-refusal cluster around vaccine questions by 12%, and either widens the threshold or adds a domain-specific allowlist. The same UI surfaces whether the under-refusal cluster moved, so the team does not trade one error for the other unknowingly. Linear is wired today; Slack, GitHub, Jira, and PagerDuty are on the roadmap.

Four-dimensional trace scoring (factual_grounding, privacy_and_safety, instruction_adherence, optimal_plan_execution, 1-5 each) is the per-trace shape; the clusters are the per-cohort shape. The Platform delivers classifier-backed evals at a lower per-eval cost than Galileo Luna-2.

Why this matters for enterprises

The application is the moral fine-tune. The lab does the base post-training and the application owns the calibration band; the deployed system is what users actually see. Three operational consequences follow.

Over-refusal rate drops on professional verticals. Customers in medical, legal, security, and finance stop opening tickets that read “the assistant refused to discuss something every human in this field discusses every day.” The threshold widens for those tenants without touching the model.

Jailbreak response time collapses. A new attack class lands on Monday. By Tuesday afternoon, the pattern is in the JailbreakScanner regex set or in a Protect adapter prompt update; by Wednesday it is deployed to every tenant whose policy includes that rule. No model release, no retraining cycle.

Compliance review stops being a quarterly bottleneck. Auditors ask “show me which inputs you block and why.” The answer is the per-tenant config plus the Error Feed clusters, not a forty-page narrative on model alignment philosophy. The control is observable, the change history is versioned, the false-positive and false-negative balance is graphed. Our LLM safety compliance guide for 2026 covers the audit framing.

SOC 2 Type II, HIPAA, GDPR, and CCPA are certified per futureagi.com/trust. ISO/IEC 27001 is in active audit.

The counterargument worth taking seriously

The fairest pushback on this position is that runtime policy is a sticking-plaster. If foundation alignment got good enough, the argument goes, the calibration problem would shrink because the model would understand context well enough to handle per-tenant variance on its own. The 2024-2025 work on system-prompt steering, in-context safety, and inference-time alignment points in that direction.

The honest read in mid-2026 is that the trend is real and the conclusion doesn’t follow. System-prompt steering and inference-time alignment are themselves runtime mechanisms; they push the calibration to a layer the deployment can configure (the system prompt), which is exactly the move we’re arguing for. The argument isn’t that the foundation model should be less capable at handling context; it’s that the policy decision belongs at a layer the tenant owns, regardless of which mechanism (system prompt, gateway guardrail, evaluator threshold) carries it.

The harder version of the counterargument is that runtime policy adds latency and a new failure surface (the guardrail itself). That is true. The honest budget is a sub-200 ms p99 latency adder for inline checks and a fail-closed default for safety-critical paths, with the regression-test suite scoring guardrail correctness alongside model correctness. Both costs are real; both are smaller than the cost of shipping the wrong calibration to every tenant.

The way forward: operationalize the tradeoff

Three takeaways for mid-2026.

1. Helpful and harmless are a Pareto frontier, not a checkbox. Stop scoring labs by which has “solved alignment.” Score by how their published frameworks handle the tradeoff, how their models behave on matched adversarial and benign sets, and how their post-training disclosures hold up under independent reproduction. Treat any vendor claim of a solved tradeoff as a sign they have gamed the helpfulness eval.
2. Measure both sides, watch the ratio. False-negative leak rate and false-positive refusal rate on the same dashboard, with the threshold movement annotated. Adversarial sets paired with matched benign sets. F1 over refusal rate.
3. Move the calibration dial out of the model. Foundation model owns the categorical floor. Application layer owns the per-tenant calibration (Protect adapters + deterministic fallbacks + per-rule action). Feedback loop owns the drift (self-improving evaluators + Error Feed clusters). Foundation alignment is one variable; the policy is another; the eval stack is the constant.

Stop asking the model to be aligned. Start asking the system to be configurable, observable, and updatable. Alignment isn’t a model property; it’s a system property, and the system is the place to fix it.

Related reading

• Frontier Model Safety Analysis (2026)
• LLM Safety and AI Regulations (2026)
• Multi-Turn Jailbreaking (Defender’s Guide 2026)

Sources

• Anthropic, Responsible Scaling Policy v3.0 (effective February 24, 2026)
• OpenAI, Preparedness Framework v2 (April 15, 2025)
• Google DeepMind, Frontier Safety Framework v3.0 (April 17, 2026)
• Bai et al., Constitutional AI (Anthropic, 2022)
• Röttger et al., XSTest: a benchmark for identifying exaggerated safety behaviors (2023, updated 2024)
• Cui et al., OR-Bench: an over-refusal benchmark for large language models (2024)
• Apollo Research, evaluation-awareness studies (2024-2026)
• Future AGI, Protect paper (arXiv 2510.13351)