MRR vs MAP vs NDCG: Retrieval Ranking Metrics in 2026

Picture a RAG retriever that scores 0.78 NDCG@10 on the eval set and 0.81 MRR. The team ships. Production faithfulness drops 9 points the next week. Investigation reveals that the retriever now puts a partially-on-topic chunk at rank 1 on 14% of queries; the generator picks up the partial chunk, the answer references the wrong policy clause, faithfulness breaks.

NDCG and MRR did not catch this because the eval set used binary relevance labels, treating partial chunks as fully relevant. The fix is to relabel the eval set with a 0 to 3 relevance scale and switch to NDCG with the graded labels. The scores were correct for the labels they were given, and those labels could not tell a partial chunk from a full one.

MRR (Mean Reciprocal Rank) scores how high the first relevant result sits, so it fits first-answer ranking tasks. MAP (Mean Average Precision) scores how early every relevant result appears, using binary labels. NDCG (Normalised Discounted Cumulative Gain) scores the whole top-k with graded relevance and a logarithmic position discount, which suits RAG retrieval.

TL;DR

If you only read one row: track NDCG@10 plus Recall@10 for RAG. NDCG captures graded ranking quality; Recall@10 catches the case where the relevant chunks are missing from the top-k entirely. For an application that uses only the top result, gate success on Hit@1 or MRR@1; keep ordinary MRR as a ranking diagnostic.

What each metric measures

Mean Reciprocal Rank (MRR)

MRR scores how high the first relevant document sits in a ranked list. For one query, the reciprocal rank is 1 / rank_of_first_relevant. If the first relevant document is at rank 1, the score is 1.0; rank 2 scores 0.5, rank 3 scores 0.33 and rank 10 scores 0.10. If no relevant document appears in the top-k, the contribution is 0.

The MRR is the mean of these reciprocal ranks across the eval set. The metric was popularised by the TREC Question Answering Track in the late 1990s and the MS MARCO passage ranking benchmark revived it for neural IR.

Mean Average Precision (MAP)

MAP rewards a system that surfaces every relevant document early. For one query, the Average Precision is computed as follows: walk down the ranked list, and at every rank where a relevant document appears, compute precision-at-that-rank. Then average those precision values across the total number of relevant documents in the corpus for that query. MAP is the mean of these Average Precision values across queries.

Take one query whose corpus has 3 relevant documents, with the ranked top-5 [rel, no, rel, no, rel]. The precisions at the 3 relevant ranks are 1/1 = 1.0, 2/3 = 0.667, and 3/5 = 0.6. The Average Precision is (1.0 + 0.667 + 0.6) / 3 = 0.756. MAP averages this across all queries.

MAP works best when relevance is binary and the corpus has a known set of relevant documents per query. It is the dominant metric in classical TREC-style IR. In modern RAG eval it is less common because relevance is usually graded.

Normalised Discounted Cumulative Gain (NDCG)

NDCG is the metric of choice when relevance is graded. Each document carries a relevance score (0 to 3 in the TREC Deep Learning track and in many RAG eval sets). Discounted Cumulative Gain at depth k is:

DCG@k = sum over i in [1..k] of: (2^rel_i - 1) / log2(i + 1)

The (2^rel_i - 1) form is the exponential gain Burges et al. used in their 2005 RankNet paper; Jarvelin and Kekalainen’s 2002 paper defined DCG with the relevance grade itself as the gain. The log2(i + 1) denominator is the position discount. Earlier ranks count more. NDCG@k normalises DCG@k by the Ideal DCG@k for that query (the DCG of a perfect ranking), so the score lives in [0, 1].

In this worked example, a query has 3 documents with relevance scores [3, 2, 1]. Suppose the system ranks them in the order [doc with rel=2, doc with rel=3, doc with rel=1]. The DCG and IDCG blocks below use the exponential gain from the formula above.

DCG@3 = (2^2 - 1)/log2(2) + (2^3 - 1)/log2(3) + (2^1 - 1)/log2(4)
      = 3/1.0 + 7/1.585 + 1/2.0
      = 3.0 + 4.417 + 0.5 = 7.917

Ideal ranking is [3, 2, 1]:

IDCG@3 = (2^3 - 1)/log2(2) + (2^2 - 1)/log2(3) + (2^1 - 1)/log2(4)
       = 7.0 + 3/1.585 + 0.5
       = 7.0 + 1.893 + 0.5 = 9.393

NDCG@3 = 7.917 / 9.393 = 0.843. The score reflects that the top-1 result was less relevant than it could have been; NDCG does not require perfect ordering, but it penalises high-relevance documents being below low-relevance ones. With linear gain the same ranking scores 0.922, so record which gain your eval uses.

What MRR, MAP, and NDCG each score

MRR cares only about the first relevant hit; MAP averages precision across all relevant hits; NDCG accounts for graded relevance.

Worked comparison of two retrievers

Suppose two retrievers return the following top-5 lists for the same query, with binary relevance labels (R = relevant, N = irrelevant), and three relevant documents exist in the corpus:

• Retriever A: [R, N, N, N, N]
• Retriever B: [N, R, R, R, N]

For MAP: AP(A) = 1.0 / 3 = 0.333 (one relevant retrieved, precision-at-rank-1 = 1.0, divided by 3 total relevant in the corpus); AP(B) = (1/2 + 2/3 + 3/4) / 3 = 0.639. For NDCG: DCG(A) = 1/log2(2) = 1.0, DCG(B) = 1/log2(3) + 1/log2(4) + 1/log2(5) ≈ 1.56, and IDCG = 1/log2(2) + 1/log2(3) + 1/log2(4) ≈ 2.13.

So MRR favours A (top-1 is correct) while MAP, Recall, and NDCG all favour B (B surfaces all 3 relevant documents inside top-5). The two retrievers have different shapes, and MRR is the metric that disagrees with the others when the top-1 is right but the rest of the top-k is wrong. Pick the metric that matches what your application reads.

How to compute MRR, MAP, and NDCG in Python

The code below reproduces the MRR, MAP and NDCG rows above with nothing but the standard library. Swap in your own relevance lists to check a library’s numbers before you trust them in CI. For library use, pytrec_eval wraps the official trec_eval tool, and scikit-learn’s ndcg_score uses linear gain, so its NDCG differs from the exponential form once your labels are graded.

import math

def reciprocal_rank(rels):
    for i, rel in enumerate(rels, start=1):
        if rel > 0:
            return 1 / i
    return 0.0

def average_precision(rels, total_relevant):
    hits, total = 0, 0.0
    for i, rel in enumerate(rels, start=1):
        if rel > 0:
            hits += 1
            total += hits / i
    return total / total_relevant

def dcg(rels, k):
    return sum((2**r - 1) / math.log2(i + 1) for i, r in enumerate(rels[:k], start=1))

def ndcg(rels, all_labels, k):
    return dcg(rels, k) / dcg(sorted(all_labels, reverse=True), k)

a = [1, 0, 0, 0, 0]
b = [0, 1, 1, 1, 0]
labels = [1, 1, 1]
for name, run in [("A", a), ("B", b)]:
    print(name, reciprocal_rank(run), round(average_precision(run, 3), 3), round(ndcg(run, labels, 5), 2))
# A 1.0 0.333 0.47
# B 0.5 0.639 0.73

When does each metric beat the others?

MRR scores first-answer ranking; Hit@1 gates top-1-only applications

When an agent executes only the first tool, use top-1 accuracy or Hit@1 to gate routing success. Ordinary MRR gives 0.5 when the correct tool ranks second, even though that tool never runs. MRR@1 also works: it is 1 for a correct first choice and 0 otherwise. Keep ordinary MRR for ranking diagnostics, FAQ retrieval, and known-item search where the first relevant answer’s position matters.

MAP is best for binary-relevance corpus search

Classical TREC ad-hoc retrieval: a query, a corpus, a known set of relevant documents per query, the system has to surface as many as possible early. MAP captures this exactly. In modern RAG, MAP is less common because the corpus is typically open-ended (no fixed relevant set per query) and relevance is graded rather than binary.

NDCG is best when relevance is graded

E-commerce search, Q&A, recommender systems, and RAG with chunk-level grading all benefit from NDCG. A 0 to 4 relevance scale lets the metric distinguish a fully-on-topic chunk from a partially-on-topic chunk. The exponential gain form penalises low-relevance documents above high-relevance ones strongly. NDCG@10 is the main retrieval metric in the BEIR and MTEB benchmarks.

Recall@k is the sanity check

None of the three ranking metrics catch the case where the relevant chunks are missing from the top-k entirely. If your retriever returns top-10 and the relevant chunk is at rank 47, MRR/MAP/NDCG all evaluate as 0 for that query but give no signal on whether the chunk exists at all. Recall@k is the companion check that asks: did the retriever even surface the relevant chunks?

Acceptable Recall@10 thresholds are domain-dependent: FAQ search, legal retrieval, code search, and medical QA all have different bars. A sustained drop in Recall@k against a stable baseline is usually the first sign the retriever needs work, regardless of the NDCG score.

How to wire these metrics into evals in 2026

The pattern is the same across platforms.

1. Build the labelled set. Enough queries to cover every intent slice you report, each with at least one relevance judgment. For graded labels, a 0 to 3 or 0 to 4 scale is standard. Sources: production traces with annotation, BEIR-style benchmark data, hand-labelled corpus.
2. Pick the depth k. Most RAG retrievers feed top-k to the generator, so k matches the production top-k (often 5, 10, or 20). Compute the metric at depth k.
3. Run the retriever in CI. Capture top-k document ids per query.
4. Compute the metric vector. MRR, MAP, NDCG@k, Recall@k, Precision@k. Store per-query and aggregate.
5. Slice the aggregate. By query intent, by query length, by user segment, by retriever component. A single aggregate hides regressions inside one slice.
6. Gate CI on a regression. Set the block threshold from the run-to-run spread of your own baseline.

OSS support varies by metric. BEIR and pytrec_eval cover classical IR metrics (MRR, MAP, NDCG@k) natively, and Phoenix computes NDCG, precision@k, reciprocal rank and hit rate on retriever spans. Ragas and DeepEval focus on RAG-quality metrics (context precision, recall, faithfulness) and typically require a thin custom scorer for MRR/MAP/NDCG. For the platform comparison, see Best RAG Evaluation Tools in 2026.

Common mistakes when using ranking metrics

• Reporting MAP on graded labels. MAP is binary; treating a 0 to 4 label as relevant-if-greater-than-zero discards information. Switch to NDCG.
• Mixing linear and exponential gain. The original DCG uses the relevance grade as the gain, and scikit-learn’s ndcg_score uses linear gain (rel / log2(i+1)); the (2^rel - 1) form punishes a relevance-1 above a relevance-3 harder. Pin one gain in the eval config and never compare scores across the two.
• Comparing MRR across systems with different k. MRR at k=10 and MRR at k=5 are different metrics. Pin k.
• Ignoring Recall@k. A retriever can have great NDCG and still miss the relevant chunks entirely; Recall@k catches this.
• One aggregate score for a heterogeneous workload. Head queries, tail queries, factual queries, opinion queries all behave differently. Slice the aggregate.
• Using the corpus’s relevance judgments without spot-checking calibration. Annotators disagree. If two annotators label the same query at 2 and 4 respectively, the metric inherits the noise. Compute IAA (inter-annotator agreement) before trusting the dataset.
• Confusing nDCG with NDCG. They are the same metric; capitalisation varies across papers and libraries. Both refer to the normalised version of DCG.
• Stopping at offline metrics. Production NDCG@10 on live traces tells you whether the retriever is still working. Wire online retrieval eval into your observability backend.

Milestones for retrieval ranking metrics

How to pick a metric for your retrieval workload

1. List what your application reads. Top-1 only, top-k, or all of them?
2. List how relevance is judged. Binary, graded, or unknown?
3. Check labelling cost. Graded labels typically cost more than binary because annotators have to assign a level rather than a yes/no; the actual multiplier varies by rubric, document length, and annotator expertise.
4. Match metric to shape. Top-1-only success -> Hit@1 or MRR@1. First-answer ranking -> MRR. Binary relevance, all of top-k matters -> MAP. Graded relevance -> NDCG.
5. Add Recall@k as the companion. Always.
6. Slice the aggregate. By intent, length, segment, component.
7. Wire it into CI. A regression on the headline metric blocks the PR.
8. Wire it into production observability. Online retrieval eval catches drift.

Keep Recall@k alongside the ranking metrics to check whether relevant results are retrieved at all.

Ranking metrics score only the retriever, so pair them with checks on what the generator does with the chunks. For depth on the broader RAG eval surface, see What is RAG Evaluation?, RAG Evaluation Metrics in 2025, and Best Rerankers for RAG in 2026.

How Future AGI scores retrieval ranking

Future AGI ships the ranking metrics from this guide as built-in evaluators: MRR, NDCG@K, Recall@K, Precision@K and Hit Rate, plus a mean_average_precision code metric. Each one compares the retrieved chunks with your ground-truth chunks and returns a 0 to 1 score with a reason, no judge model involved. Run them next to Groundedness, Context Relevance and Chunk Attribution so retrieval and generation are scored on the same dataset.

For graded labels, the local ndcg metric in agent-learning-kit (pip install agent-learning-kit) takes a relevance grade per chunk and uses the same (2^rel - 1) gain as the worked example above. In production, continuous eval tasks score a sampled share of new spans, and alerts fire when an eval score crosses a fixed threshold or drops a set percentage from its baseline.

traceAI (Apache 2.0) instruments LangChain, LlamaIndex, Pinecone, Qdrant, Weaviate, ChromaDB and Milvus on OpenTelemetry, so each retrieval step lands as a span you can score. The Agent Command Center gateway routes traffic across 100+ providers with guardrails inline. The Future AGI platform is built on an Apache-2.0 core you can self-host, and pricing starts free with 50 GB of tracing storage a month.

Catching the partial-chunk regression before it ships

Go back to the retriever that shipped at 0.78 NDCG@10 and lost 9 points of faithfulness. With graded labels on a 0 to 3 scale, a partially-on-topic chunk at rank 1 earns less gain than a fully on-topic one, so NDCG@10 drops when the retriever makes that swap. A CI gate on graded NDCG@10, plus Recall@10, turns that drop into a blocked pull request instead of a faithfulness incident.

Gating a retriever on MRR and NDCG? Start free on Future AGI and score your first retrieval eval set.

Sources

• Introduction to Information Retrieval (Manning, Raghavan and Schütze)
• Cumulated Gain-based Evaluation of IR Techniques (Järvelin and Kekäläinen, 2002)
• BEIR: Heterogeneous Benchmark for Zero-shot IR
• MTEB leaderboard