What Is a Custom LLM Judge Metric?

What Is a Custom LLM Judge Metric?

A custom LLM judge metric is an evaluation metric you define by writing a rubric and wrapping a judge model around it. The judge reads the input, the candidate output, and any retrieved context, applies your scoring rules, and returns a numeric or categorical score plus a reason. Custom judges cover the gap built-in evaluators miss — domain-specific correctness, brand-voice fit, regulated-industry policies, internal style guides. They are essential when no off-the-shelf metric matches the task, and they demand careful prompt design, calibration against human labels, and ongoing drift checks.

Why It Matters in Production LLM and Agent Systems

A built-in AnswerRelevancy score does not know your style guide. A built-in Groundedness score does not know your industry’s compliance rules. If you ship a customer-facing chatbot for a regulated insurer, “policy-compliant phrasing” is a metric only your team can write — and without it, the model can be fluent, grounded, and still legally non-compliant. The custom judge is how you turn an internal rubric into a number that lives next to BLEU and Faithfulness on the dashboard.

The pain hits across roles. A product team wants to enforce that a financial advice agent never says “guaranteed return” — a built-in judge has no concept of that prohibition. A clinical-content team needs every diabetes answer to cite the correct ICD code; off-the-shelf factuality won’t catch a wrong code. A brand team needs every marketing email written by the LLM to match a tone-of-voice rubric; tone is too domain-specific for a generic helpfulness score.

In 2026 agent stacks, custom judges become more valuable, not less. Every domain agent ships with a domain-specific evaluation surface. The risk is laziness — teams write a one-line judge prompt, never calibrate it, and treat its output as truth. A custom judge that has not been compared against human labels is just an LLM call dressed up as a metric. Calibration discipline is what makes the metric trustworthy.

How FutureAGI Handles Custom LLM Judge Metrics

FutureAGI’s approach is to make custom judges first-class through the CustomEvaluation class in fi.evals. You define the judge as a callable that receives input/output/context, returns a score and reason, and registers metadata — name, prompt template, judge-model identifier. The evaluator runs anywhere a built-in evaluator runs: offline through Dataset.add_evaluation(), online through traceAI sampled spans, or as a Guard post-guardrail check.

Calibration is built into the workflow. You provide a labelled subset of the Dataset (human-graded rows), and the platform reports per-judge agreement metrics — accuracy, Cohen’s kappa, per-class confusion. If agreement falls below threshold, the prompt is rewritten or the judge model swapped. For ongoing drift, judge agreement is re-checked weekly against a fresh labelled cohort, and a regression alert fires if it slips. The optimizer surfaces — ProTeGi, MetaPromptOptimizer, PromptWizardOptimizer — can also iterate the rubric prompt itself, turning a hand-written rubric into one tuned against a calibration set.

Compared to writing a one-off scoring prompt in a notebook, the FutureAGI path turns the custom judge into a versioned, calibrated, dashboarded metric — the only form that survives contact with production. Compared to G-Eval-style structured-rubric judges, CustomEvaluation plays the same role but ties cleanly into the eval and trace layers your team already uses.

How to Measure or Detect It

A custom judge is a metric, but the judge itself also needs to be measured:

• CustomEvaluation: the canonical class for wrapping a rubric as a callable evaluator.
• Judge-human agreement: Cohen’s kappa or accuracy of the judge versus human labels on a held-out cohort — the trust score for the judge itself.
• Per-rubric-criterion score: if the rubric has multiple criteria, return per-criterion scores for richer dashboards.
• Eval-fail-rate-by-cohort (dashboard signal): aggregated custom-judge fail rate, sliced by route, model, or user cohort.
• Judge cost-per-trace: a custom judge running on every trace adds inference cost — measure and budget it.
• Drift in agreement: weekly re-calibration delta; alert if kappa drops more than X.

Minimal Python:

from fi.evals import CustomEvaluation

policy_judge = CustomEvaluation(
    name="brand_voice_compliance",
    judge_model="gpt-4o",
    prompt="""Score 0/1: does the response use the brand's neutral tone
    and avoid the banned phrase 'guaranteed return'?
    Input: {input}
    Output: {output}
    Return JSON: {"score": 0|1, "reason": "..."}""",
)

result = policy_judge.evaluate(input=user_q, output=model_resp)
print(result.score, result.reason)

Common Mistakes

• Skipping calibration. A custom judge with no human-label comparison is opinion, not measurement. Always report agreement on a held-out cohort.
• Using the same model as judge and generator. Self-evaluation inflates scores. Pin the judge to a different model family or provider.
• Vague rubrics. “Score 0–10 for quality” gives noisy outputs. Decompose the rubric into specific criteria with concrete pass/fail anchors.
• No drift monitoring. Judge models change behaviour with provider updates. Re-calibrate weekly or on every provider model bump.
• Running expensive judges on every trace without a budget. Sample first, score everything later — or budget the cost up front.