Evaluating DeepSeek Models in 2026: Capability Shape, English Transfer, Safety, and Residency

A team in Singapore ships a research assistant on DeepSeek R1 because pricing is six percent of Claude Opus and the reasoning demo lands. By the third week, the auditor flags three failures. Fourteen percent of answers contradict the chain-of-thought the model just wrote. The Chinese-language route throws safety-block rates six times higher than the English route on identical prompts because the classifier was trained on English. The quantized Q4 build the platform team rolled to save GPUs disagrees with the FP16 reference on roughly one in twelve hard cases. None of these show up in a rubric ported from a GPT-5 agent, because that rubric never scored a reasoning trace, never stratified by language, and never treated quantization delta as a first-class axis.

DeepSeek changed the cost curve of frontier reasoning in early 2025. V4 in 2026 closed the gap further: it matches frontier on the SWE-bench Verified slice vendors report and ships open weights under a permissive license. The cost-and-capability story is real. The safety, residency, and English-versus-Chinese gap require an honest eval that closed-frontier evaluation never had to think about.

The opinion this post earns: DeepSeek’s open-weight and cost story is real, and the safety, residency, and English-task gap require honest measurement that public leaderboards will not give you. The methodology has four gates, all of them. Capability-shape benchmarks where DeepSeek shines, paired comparison on YOUR English data against the incumbent, a safety regression check, and a residency-and-license gate. Pass all four and DeepSeek substitutes cleanly. Pass three and you ship a quality cliff dressed as a cost win.

This guide is the working playbook for evaluating DeepSeek V3, R1, and V4 as production substitutes for Claude Sonnet 4.5 and GPT-5. The methodology is code-defined against the ai-evaluation SDK, wired through the Agent Command Center for shadow traffic and residency routing, and shaped by the arena-judge pattern on the paired-comparison side.

TL;DR: the four-gate scorecard

Substitute when all four pass. Three out of four is a quality cliff dressed as a cost win.

Why DeepSeek evaluation is its own discipline

DeepSeek’s API looks like OpenAI’s at first glance. Same /v1/chat/completions endpoint, same message shape, same tool-call schema in V3 and V4. Drop a base_url swap and the calls go through. The differences emerge under load, under audit, and under the buyer’s compliance review.

The reasoning trace is a separate artifact. DeepSeek R1 emits a reasoning_content field alongside the final content. The trace is not hidden internal state; it is separately addressable for your eval to read, score, and store. Most eval suites built for GPT-5 or Claude ignore it and miss the failure mode where the trace is correct and the answer is wrong. The LLM benchmarks vs production evals work shows this happens on eight to fifteen percent of R1 traces on hard reasoning sets. An answer-only rubric is half the diagnostic value of running a reasoning model.

Weights are open and quantization is the production shape. Teams self-host V3, R1, and V4 on vLLM, SGLang, or TGI in Q4, Q5, or Q8. Quantization changes behavior in ways benchmark scores hide. The FP16 reference and the Q4 build agree on ninety-plus percent of cases and disagree on the ten percent that matter. A defensible eval runs the same prompt set through both and scores the delta.

The buyer surface is geopolitically loaded. DeepSeek’s hosted API egresses to the People’s Republic of China. Regulated buyers in the US, EU, India, and parts of Southeast Asia cannot route customer data there. The remediation is self-hosted weights behind a gateway in your VPC, which only works if your eval can prove no traffic leaves the perimeter. License terms on V3, R1, and V4 are open but constrained; legal needs the read on file before the first regulated tenant ships.

The English-versus-Chinese capability gap is real and uneven. DeepSeek’s bilingual training is a strength; the capability split on long-tail tasks is a measurement problem. An English-only golden set hides regressions in Chinese that bilingual users notice on day one. A safety-block rate measured only in English under-represents false positives Western-trained classifiers throw on Chinese content. Run every rubric stratified by language.

Gate 1: capability-shape benchmarks where DeepSeek shines

Running thirteen public benchmarks is theatre. Pick the two or three that map to DeepSeek’s strengths and the failure mode your application cannot afford. Three families earn the budget; the rest are decoration.

Math and symbolic reasoning (GSM8K, MATH). DeepSeek R1 and V4 sit at frontier-parity on GSM8K and within two to four points of closed frontier on MATH. If your application does multi-step arithmetic, equation manipulation, or chain-of-thought planning, this is where DeepSeek’s reasoning training pays off. Score on a held-out 200-question subset; the headline leaderboard number is contaminated.

Code generation and software engineering (HumanEval, SWE-bench Verified). HumanEval is largely saturated; treat it as a smoke test. SWE-bench Verified is the harder signal: end-to-end software-engineering tasks across real repositories with verified passing tests. DeepSeek V4-Preview reports SWE-bench Verified scores at frontier-parity; run your own subset because those scores have not been independently reproduced at scale. The evaluating coding agents guide covers the harness.

Tool composition (BFCL or custom probe). DeepSeek V3 and V4 ship OpenAI-compatible tool calling. Single-tool calls work; composition is the cliff. Open-weight models often match flagship on single-tool calls and drop four to eight points on chains where the agent picks tool A, reads the result, and decides tool B with arguments derived from A’s output. If your application uses two or more tools per turn, build 50 to 100 multi-tool chains from production traces with expert-labeled correct sequences. The evaluating tool-calling agents post covers the composition failure shape.

Skip aggregate leaderboards as a substitution gate. MMLU averages 57 subjects; on the four that matter to your application, DeepSeek can drop ten points and still post a flagship-adjacent overall.

from fi.evals import Evaluator
from fi.evals.templates import (
    FactualAccuracy, TaskCompletion, EvaluateFunctionCalling,
)
from fi.testcases import TestCase

evaluator = Evaluator()  # FI_API_KEY / FI_SECRET_KEY from env

CAPABILITY_RUBRICS = {
    "math":  [FactualAccuracy()],
    "code":  [TaskCompletion()],
    "tools": [EvaluateFunctionCalling()],
}

def shape_score(shape, samples, model_fn):
    return evaluator.evaluate(
        eval_templates=CAPABILITY_RUBRICS[shape],
        inputs=[
            TestCase(
                input=ex.input,
                output=model_fn(ex.input),
                context=getattr(ex, "context", ""),
                expected_output=getattr(ex, "gold", None),
            )
            for ex in samples
        ],
    ).eval_results

Run against DeepSeek and against the incumbent on the same week, same hardware path. The per-axis delta is the answer. A candidate that loses six points on EvaluateFunctionCalling and ties on FactualAccuracy is a candidate for chat workloads, not agent workloads.

Gate 2: paired comparison on YOUR English data

Capability-shape benchmarks tell you where DeepSeek is theoretically competitive. They do not tell you whether it transfers to your application. The benchmark that does is paired comparison: send the same input to incumbent and candidate simultaneously, capture both responses, hand the pair to a third-party arena judge with position randomized, and ask which is better. Aggregate winrate against the incumbent cancels rubric drift, neutralizes input-distribution shifts, and matches the way humans pick a winner.

Five details separate a working arena gate from one that flatters.

• Sample from production English data. 200 to 500 inputs the model would actually see, stratified by intent, length, and difficulty. The English-versus-Chinese gap on DeepSeek is real; do not paper over it with a mixed-language sample. Run an English gate, then a separate Chinese gate if the workload is bilingual.
• Randomize position per pair. Judges have a 10-15 point position bias on close calls. Flip the order on every comparison and the bias cancels.
• Judge from a different model family. Sonnet judging GPT against DeepSeek is fine; DeepSeek judging itself is not. Same-family judging inflates self-preference by five to eight points.
• Report wins, losses, and ties separately. A 58/12/30 split is not the same model as 58/40/2 at matched winrate. High tie rates mean the candidate is indistinguishable from the incumbent on those inputs.
• Bound the verdict by sample size. The 95 percent CI on a winrate p with n pairs is roughly ±1.96 × sqrt(p × (1 - p) / n). At p = 0.50 and n = 200 the interval is ±6.9 points, which crosses the substitution line and decides nothing. At n = 500 it narrows to ±4.4. Run the power calculation before wiring the gate.

The arena gate as code, against the CustomLLMJudge primitive:

from fi.evals import Evaluator
from fi.evals.metrics.llm_as_judges.custom_judge.metric import CustomLLMJudge
from fi.evals.metrics.llm_as_judges.types import CustomInput
from fi.evals.llm.providers.litellm import LiteLLMProvider
import random

arena_judge = CustomLLMJudge(
    provider=LiteLLMProvider(),
    config={
        "name": "deepseek_vs_incumbent",
        "model": "claude-sonnet-4-5-20250929",
        "grading_criteria": (
            "Compare two responses to the same input. "
            "Optimize for helpfulness, accuracy, tool-call correctness, "
            "and adherence to the requested format. Do not prefer longer answers. "
            "Return 1.0 if ANSWER_A is better, 0.0 if ANSWER_B, 0.5 if tie."
        ),
    },
)

def paired_winrate(samples, incumbent_fn, candidate_fn, n=300):
    wins = losses = ties = 0
    for ex in random.sample(samples, min(n, len(samples))):
        inc, cand = incumbent_fn(ex.input), candidate_fn(ex.input)
        flip = random.choice([True, False])
        ans_a, ans_b = (cand, inc) if flip else (inc, cand)
        out = arena_judge.compute_one(CustomInput(
            question=ex.input, answer_a=ans_a, answer_b=ans_b,
        ))["output"]
        if out == 0.5:
            ties += 1
        elif (out == 1.0 and flip) or (out == 0.0 and not flip):
            wins += 1
        else:
            losses += 1
    return {"wins": wins, "losses": losses, "ties": ties}

Substitution-ready winrate floor: 0.48 on 300+ pairs is a candidate worth substituting if the cost win is large; 0.52 is a clear go; 0.45 is a regression dressed as a tie. Decide the floor before the run. For R1, run a third judge that takes the reasoning trace and the final answer together and scores whether the answer follows from the trace. A good trace with a wrong answer is the R1-specific execution failure that answer-only evals miss.

Gate 3: safety regression check

DeepSeek’s safety training is lighter than the closed US frontier, and the regression goes both ways. Jailbreak attempts succeed at higher rates on R1 and V3 than on Claude or GPT-5. Refusal calibration drifts toward over-refusal on medical-adjacent, legal-adjacent, and security-research queries the incumbent answers cleanly. Both are release blockers.

Run four checks against the incumbent and the candidate on the same payloads.

• Prompt injection (OWASP LLM01). Fixed payload set from Garak or PromptInject plus a domain-specific custom set. Score with the PromptInjection template. A higher compliance rate on the candidate is a release blocker.
• Jailbreak attempts. Fixed harmful-instruction suite. The red-teaming step-by-step guide covers the payload set worth running.
• Refusal calibration. A stratified set with ground-truth labels for should_answer versus should_refuse. Score with AnswerRefusal plus a CustomLLMJudge rubric for over-refusal severity. DeepSeek drifts toward over-refusal on medical-adjacent and legal-adjacent queries; that failure mode does not show on any accuracy benchmark.
• System-prompt leakage. Probe the candidate to leak the system prompt verbatim; compare leakage rate to the incumbent.

For Chinese output, the safety stack matters as much as the model. The ai-evaluation SDK ships guardrail backends including the QWEN3GUARD family (8B, 4B, 0.6B) that produces lower false-positive rates than Western-trained classifiers like LlamaGuard on Chinese content. Wire a bilingual aggregation strategy at the output rail so production matches the offline gate.

from fi.evals import Guardrails
from fi.evals.types import RailType, AggregationStrategy
from fi.evals.guardrails import LLAMAGUARD_3_8B, QWEN3GUARD_8B

bilingual_safety = Guardrails(
    rail_type=RailType.OUTPUT,
    aggregation=AggregationStrategy.MAJORITY,
    backends=[LLAMAGUARD_3_8B, QWEN3GUARD_8B],
)

The release rule is sharp. Any regression on the refusal or injection axes is a blocker. Parity or better, or the candidate does not ship. Future AGI Protect runs the same checks inline at 65 ms median time-to-label, so the offline rubric and the production guardrail use the same adapters.

Gate 4: the residency, license, and compliance gate

This gate separates a working DeepSeek deployment from a compliance incident. The hosted API at api.deepseek.com egresses to the PRC. Regulated buyers in healthcare, finance, education, and government across the US, EU, India, and parts of Southeast Asia cannot route customer data there. The remediation is self-hosted weights behind a gateway in your VPC, and the eval has to prove no traffic leaves the perimeter.

Three artifacts make the gate auditable.

Egress proof. Route every DeepSeek call through the Agent Command Center deployed in your VPC (single Go binary, Apache 2.0, 100+ providers, 18+ built-in guardrail scanners). Configure it to refuse api.deepseek.com as a backend on the regulated tenant and route only to the self-hosted vLLM cluster. Assert on the x-agentcc-model-used header in CI so a config change cannot ship a regulated tenant onto the hosted API.

License review on file. DeepSeek-V3, R1, and V4 ship under the DeepSeek License: permits commercial use, adds restrictions worth a legal read end to end. More permissive than Llama 2’s RAIL, more restrictive than Apache 2.0. Your buyer will demand a signed legal review, not a tweet-thread interpretation.

Per-tenant audit trail. Every span carries llm.model_name, the provider, the gateway-attributed cost, and the inferred input language. Sample nightly and assert on the model-attribution allow-list per tenant. The self-hosted gateway alternatives post covers the deployment-mode tradeoffs.

The gateway returns per-call cost and model attribution as response headers. That is the table you route from once the four gates are green:

import requests

response = requests.post(
    "https://gateway.futureagi.com/v1/chat/completions",
    headers={"Authorization": f"Bearer {FAGI_KEY}"},
    json={"model": "deepseek-v3", "messages": messages},
)

cost = float(response.headers["x-agentcc-cost"])
latency_ms = float(response.headers["x-agentcc-latency-ms"])
model_used = response.headers["x-agentcc-model-used"]

Sticker cost is dollars per million tokens. Effective cost is dollars per accepted output after retries on parse failures, refusal misfires, schema violations, and fallback to incumbent on quality misses. DeepSeek’s unit cost can be ninety-five percent below the incumbent and still produce a worse effective cost if the candidate fails the quality gate twenty-five percent of the time. The LLM eval cost optimization work covers the routing math.

The production rollout: canary, not big-bang

Pass all four gates and substitution is ready. The rollout is canary.

1. Route 5 to 10 percent of production traffic to DeepSeek through the gateway; the rest stays on the incumbent.
2. Attach the same rubrics that ran in offline gates as span-attached scorers on live traces via traceAI and EvalTag. Per-rubric scores live next to latency, model, and cost on every OpenTelemetry span.
3. Sample paired requests through gateway shadow mode and run the arena judge on the pairs. Accumulate winrate over a rolling 30-60 minute window.
4. Alarm on a 2-point drop in any per-rubric rolling mean or a winrate drop below the agreed floor. Auto-rollback the canary cohort if the alarm sustains.

The Agent Command Center handles the canary split with eval-gated rollback across 100+ providers. Shadow, mirror, and race modes are configured by header; none require app-code changes once the gateway base URL is set. The shadow traffic and canary patterns post covers the rollout side.

Keep the rubric pinned. The moment the CI gate and the canary disagree, the dataset stopped being representative; promote the failing canary traces back into the offline set and rerun the four gates. That is the closed loop.

How Future AGI ships DeepSeek evaluation

Future AGI ships the eval stack as a package. Start with the SDK and the arena-judge primitive for code-defined gates. Graduate to the Platform when you want self-improving rubrics and per-cluster failure routing.

• ai-evaluation SDK (Apache 2.0). 50+ EvalTemplate classes covering the four gates: Groundedness, ContextAdherence, FactualAccuracy, TaskCompletion, EvaluateFunctionCalling, AnswerRefusal, PromptInjection, DataPrivacyCompliance. CustomLLMJudge is the primitive for the three reasoning-specific rubrics R1 needs (trace quality, trace-answer alignment, multilingual parity). Local heuristic metrics run offline at sub-second latency.
• traceAI. 50+ AI surfaces across Python, TypeScript, Java, and C#. Every DeepSeek call lands as an OpenTelemetry span with llm.model_name set; a custom span processor captures reasoning_content as deepseek.reasoning_tokens so the trace is available to the offline rubric.
• Agent Command Center. Single Go binary, Apache 2.0, 100+ providers, 18+ built-in guardrail scanners. Shadow, mirror, and race modes for paired traffic. VPC-deployable for the residency gate. Returns x-agentcc-cost, x-agentcc-latency-ms, x-agentcc-model-used, x-agentcc-fallback-used on every call.
• Future AGI Protect. Four Gemma 3n LoRA adapters (toxicity, bias_detection, prompt_injection, data_privacy_compliance) at 65 ms median time-to-label per arXiv:2510.13351. The same adapters power the offline rubric and the inline guardrail so test and production policy stay in sync.
• Future AGI Platform. Self-improving evaluators that retune from production feedback at lower per-eval cost than Galileo Luna-2. Error Feed clusters failing traces with HDBSCAN and a Sonnet 4.5 Judge writes the immediate_fix per cluster. Common DeepSeek clusters: R1 trace correct but answer wrong, V3 tool selection drift past five tools, quantized variant disagrees with FP16 on edge cases, Chinese output triggers a Western-trained classifier false positive. SOC 2 Type II, HIPAA, GDPR, CCPA certified; ISO/IEC 27001 in active audit.

Drop ai-evaluation and the arena-judge primitive into the substitution gate this afternoon. Add traceAI and the gateway shadow mode when the candidate is ready for canary. Turn the Platform and Error Feed on when per-cluster routing becomes the bottleneck.

Ready to evaluate your first DeepSeek substitution? Run pip install ai-evaluation, scaffold the four gates against your golden set, point the gateway at https://gateway.futureagi.com/v1 for shadow traffic or deploy the Go binary in your VPC for the residency gate. The DeepSeek route that clears all four checks is a real cost win; everything else is a quality cliff the leaderboard didn’t show you.

Related reading

• Evaluating Cheap Frontier Models in 2026
• LLM Arena as a Judge: Pairwise Comparison Evals (2026)
• Evaluating Tool-Calling Agents in 2026
• Best vLLM Self-Hosted Inference Alternatives (2026)
• LLM Benchmarks vs Production Evals in 2026
• Red Teaming LLMs: A Step-by-Step Guide (2026)
• LLM Eval Shadow Traffic and Canary Patterns (2026)
• Best Self-Hosted AI Gateways (2026)
• LLM Eval Cost Optimization (2026)
• The State of LLM Benchmarking (2026)