What Is Named Entity Recognition: How to Measure NER Quality in LLM Pipelines

Entity extraction is the quietest part of most LLM pipelines. It rarely errors, it always returns something, and the something is often subtly wrong.

A model reads an invoice and returns Acme Corp. The document says Acme Corporation Ltd.. Nothing failed. No exception, no retry, no red span in your trace. Downstream, the record does not match the customer table, and a human spends twenty minutes finding out why.

That is the shape of an NER quality problem in 2026. The extraction step works, the output parses, and the error only surfaces three systems later. This post covers what named entity recognition is, how it changed once LLMs took over the task, and how to score it properly with precision, recall, F1, and partial matching.

What Is Named Entity Recognition?

Named entity recognition is the task of finding spans of text that refer to real-world things and labeling each one with a type. A span is a stretch of characters in the source. A label says what kind of thing it is. That is the whole job.

It sits underneath a lot of software you already use. Résumé parsers pull employer names. Contract tooling pulls parties and dates. Support systems pull order IDs. Any pipeline that turns prose into structured fields is doing NER, whether or not anyone calls it that.

Core Entity Types

Most schemas start from the same small set. Person, organization, location, date, and money cover the bulk of business documents. Teams then add domain types on top: drug names in healthcare, ticker symbols in finance, part numbers in manufacturing.

The type set is a design decision, not a given. A tighter schema is easier to score because there is less room for two reasonable annotators to disagree. A sprawling one produces disputes about whether the Berlin office is a location or an organization.

A Worked Example

Take one sentence and tag it:

Maria Chen (PERSON) signed with Northwind Logistics (ORG) in Rotterdam (LOC) on March 3, 2026 (DATE) for $1.2M (MONEY).

Five entities, five types, five exact spans. This sentence is the reference point for every metric in this post. Hold onto it, because the numbers later come from scoring an imperfect model against exactly these five.

How Named Entity Recognition Works, From Rule-Based Systems to LLMs

The task has stayed the same for two decades. The machinery underneath has been replaced twice.

The Traditional NLP Pipeline

Classic NER is a chain of narrow steps. Tokenization splits text into tokens. Part-of-speech tagging labels each token grammatically. Chunking groups tokens into phrases. A sequence model then assigns an entity tag to every token, usually in BIO format: B-ORG for the first token of an organization, I-ORG for tokens inside it, O for everything else.

The useful property here is structure. Every prediction is anchored to a token index, so the span boundaries are exact by construction. You always know precisely which characters the model claimed. Scoring is mechanical.

The cost is rigidity. The model only knows the types it was trained on, and adding a new one means labeling data and retraining. Ambiguity is also hard, since the tagger has no real notion of what a sentence is about.

How LLM-Based Extraction Differs

An LLM does none of that. You describe the entity types in a prompt, and the model writes back a list, usually JSON. There is no tagging step and no token index. The output is generated text that happens to look like structured data.

This buys enormous flexibility. New entity types cost a prompt edit instead of a training run. Ambiguous cases get resolved with actual context, which is where taggers struggle most.

It also breaks the guarantee. The model is producing text, not pointing at text. Northwind Logistics can come back as Northwind, or Northwind Logistics B.V., or an entity that never appeared in the document. Nothing in the generation process forces the output to be a substring of the input.

A 2025 pilot benchmark makes the tradeoff concrete. Testing six systems on ambiguous entities, the authors found LLMs averaged 0.692 macro-F1 against 0.655 for traditional tools, with Gemini-1.5-flash top at 0.824, a traditional tagger (Stanza) close behind at 0.806, and spaCy at 0.741 (arXiv:2509.12098).

The within-family spread was wider than the gap between families: Qwen-3-4B scored 0.527 and NLTK scored 0.416. It is a small pilot on a 119-token dataset, so treat it as directional. The direction is that neither family dominates.

That same paper flags a scoring caveat worth carrying forward. Its token-level evaluation “may penalize minor boundary errors (e.g., one missing token inside a multi-word organization), slightly underestimating true span-level F1.” How you count a near-miss changes the score materially.

Why Measuring NER Quality Is Different in LLM Pipelines

Classic NER metrics assume the prediction is a token range. Once the prediction is free text, two failure modes appear that the old metrics were never built to catch.

Hallucinated Entities

The model returns an entity that is not in the source. Sometimes it is a plausible completion, like expanding an abbreviation into a full company name that was never written. Sometimes it is drawn from the prompt’s own examples.

This is a precision failure with an unusual property: it is invisible to a ground-truth comparison in the one case that matters most. If the hallucinated value happens to match the expected answer, a lenient scorer marks it correct. It was still not read from the document.

The direct test is containment. Every extracted entity string should appear in the source text, and a simple substring check catches fabrications that were never in the document at all, before any semantic scoring runs. Cheap, deterministic, and worth running first. For the broader class of this problem, we have a full breakdown of hallucination detection methods.

Span and Formatting Drift

The subtler failure is reformatting. The model finds the right entity and then rewrites it. March 3, 2026 becomes 2026-03-03. $1.2M becomes 1200000. Acme Corporation Ltd. becomes Acme Corp.

Every one of those is a correct read and an exact-match failure. Score them strictly and your F1 collapses while your extraction is actually fine. Score them loosely and you stop noticing when the model trims a genuinely load-bearing part of a name.

Normalization drift is worse in high-stakes fields because it is so easy to accept. A date rewritten into ISO format looks tidier than the source. That tidiness is the model editing your data.

Core Metrics for Entity Extraction Quality

Three numbers do most of the work. They are the same three used in classic NER, and they still apply, as long as you are explicit about what counts as a match.

Precision, Recall, and F1

Go back to the worked sentence with its five true entities. Say the model returns four: Maria Chen, Northwind Logistics, Rotterdam, and Global Freight (which is not in the text). It missed the date and the money value.

Precision asks how many extractions were correct: three out of four, so 0.75. Recall asks how many true entities were found: three out of five, so 0.60. F1 is the harmonic mean, which lands at 0.67.

Which number you optimize depends on the cost of being wrong. In a redaction pipeline, a missed entity is a leak, so recall dominates. In an auto-populated CRM field, a wrong entity corrupts a record, so precision dominates. Report both, always, and treat a lone F1 as a summary rather than an answer.

Also score per entity type. An aggregate F1 of 0.90 can hide a MONEY class sitting at 0.55 if that class is rare. The same trap shows up in PII detection evaluation, where one weak category is the entire risk.

Exact Match vs Partial Match

The match rule decides everything above. Exact match requires the predicted string and its boundaries to be identical to the reference. It is unforgiving, deterministic, and easy to defend in an audit.

Partial or fuzzy match credits overlap. Northwind against Northwind Logistics scores as a hit, or as partial credit depending on your scheme. It reflects how LLMs actually behave, at the cost of a judgment call baked into every score.

Run both. The gap between your exact-match F1 and your partial-match F1 is a direct measurement of formatting drift. If exact is 0.62 and partial is 0.91, the model is finding your entities and rewriting them, and that is a prompt problem, not a capability problem.

How Do You Build an NER Evaluation Pipeline for LLM Output?

Four pieces, in order: a labeled reference set, a match rule, a scorer that runs on every change, and per-type reporting. None of it is exotic. Most teams skip it because extraction rarely throws errors.

Ground-Truth Comparison and Fuzzy Matching

Start with the reference set. A few hundred documents, hand-labeled with spans and types, drawn from your real distribution rather than clean samples. Include the ugly ones: OCR noise, tables, abbreviations, multilingual names. Those are where extraction fails.

Then fix the match rule before you look at any scores. Decide what happens to case differences, trailing punctuation, legal suffixes, and normalized dates. Write it down. A match rule chosen after seeing the numbers is not a measurement.

A practical layering works well. Exact match first, since it is free. Normalized match second, after lowercasing and stripping suffixes. Fuzzy or semantic match last, only for the residue, because it is the expensive and least deterministic tier.

Custom Evals for Extraction Correctness at Scale

Manual spot-checking does not survive contact with volume. Once you have a rule, it needs to run automatically on every prompt change, model swap, and provider update, and to fail loudly when a score moves.

That is an eval, and it belongs in CI next to your unit tests. The design principles are the same ones that apply to any custom LLM eval metric: a fixed dataset, a deterministic rule where possible, and a threshold that fails the build.

Watch the entity-type breakdown over time, not just the headline number. Aggregate F1 is remarkably stable while individual classes degrade, particularly rare ones. A per-type table catches the regression that a single score smooths away.

Future AGI

Be clear about the boundary first: Future AGI ships no NER benchmark dataset, and no span-level scorer that understands BIO tags and scores per entity type. There is no button labeled “NER.” What it does ship is most of the pipeline above as built-ins, with custom evals for the rest. Here is what maps to what.

Start with the metrics themselves. The Statistical & Classification Metrics evaluator is a code-based check that scores predicted labels against expected ones, and it exposes f1_score, precision_score, and f_beta_score directly. You do not write the harmonic mean yourself. What you write is the part that turns entity spans into the labels those metrics compare.

Custom evals are the core piece. You define an eval by name, pick a type (Agent Evaluator, LLM-as-Judge, or Code Eval), write instructions specifying what counts as a pass or a fail, and choose an output type of Pass/fail, Scoring, or Choices (custom evals docs). A Code Eval is the natural home for your containment check and exact-match logic.

Once saved, the docs note it “works exactly like a built-in: use it on a dataset, a trace, a simulation, or from the SDK, by its name” (custom evals docs). That is what turns a scoring script into something that runs on every change.

Ground truth handles the reference side. You upload an annotated dataset as CSV, Excel, or JSON up to 50 MB and map eval template variables to columns. At run time the eval “retrieves the most similar rows and injects them into the evaluator prompt as calibration examples” (ground truth docs). Your labeled entity set becomes calibration for the judge rather than a static file.

For the match tiers, the built-in “Reference & similarity” group already covers ground. Ground Truth Match is an LLM-as-Judge eval comparing a generated_value against an expected_value, and it “judges whether they are equivalent in meaning, not just in wording,” returning Pass/Fail with a reason (ground truth match docs).

Fuzzy Match is a second LLM-judged tier, sitting alongside Ground Truth Match rather than below it. It compares output against an expected answer “using approximate text matching that tolerates minor wording, spelling, or formatting differences,” and returns a numeric score rather than a verdict (fuzzy match docs).

The docs also note it can be less reliable on very short text, and entity spans usually are short, so treat it as a signal to combine with the others rather than a standalone verdict. For the normalized tier, the docs’ own answer to that short-text caveat is Levenshtein Similarity, a deterministic character-edit metric that scores similarity by the minimum number of single-character edits required. Between Ground Truth Match, Fuzzy Match, and Levenshtein Similarity you get exact, semantic, and character-level matching without writing any of the three yourself.

Detect Hallucination covers invented entities. It “identifies fabricated facts or details a model added that aren’t present in the input or provided context,” taking the model output and the context and returning Pass/Fail plus a plain-language reason (detect hallucination docs). Point context at the source document and it becomes a semantic backstop behind your substring check.

There is also a built-in PII eval that “scans text for personally identifiable information such as names, addresses, or ID numbers” slipping into responses (PII docs). It is entity detection with a compliance framing rather than an extraction-quality framing, so it complements an NER eval rather than replacing one.

The honest summary: you assemble precision, recall, and F1 for entity extraction from custom evals plus ground-truth matching. The components are there and the scoring logic is yours to define.

Conclusion

NER quality in an LLM pipeline is not the problem it was five years ago. A token-level tagger could not invent an organization or rewrite a date into ISO format. A generative model does both routinely, and neither shows up as an error.

So the metrics need adjusting rather than replacing. Precision, recall, and F1 still work. What changes is that you have to decide what a match means before you compute them, run a containment check for fabricated entities, and report the gap between exact and partial scoring instead of picking one.

Start with a few hundred hand-labeled documents from your real traffic. Write the match rule down before you see a score. Then automate it, so the next prompt tweak that quietly breaks your date extraction fails a build instead of a customer record.