Evaluative Mechanics: Demystifying Retrieval Metrics for RAG

The Retrieval Evaluation Pipeline

In Retrieval-Augmented Generation (RAG), the performance of your system is strictly capped by the quality of its retrieval engine. If the retrieval step feeds irrelevant context to the Large Language Model (LLM), the generator will hallucinate, lose context, or generate garbage output.

While vector databases use raw distance metrics like Cosine Similarity or L2 distance to index and retrieve vectors, these metrics do not tell you whether the retrieved documents are actually correct or well-ordered for downstream tasks. To evaluate this, you must use traditional Information Retrieval (IR) metrics.

The evaluation pipeline takes a query, retrieves the top \(K\) documents from a vector index, and compares them against a gold-standard ground truth. The ground truth consists of the set of known relevant documents in your database.

Query Q Vector Store (N) KNN Index (Top K) Top K Stack Evaluation Layer • Precision & Recall • Hit Rate @ K • Reciprocal Rank • Average Precision • NDCG (Graded)
The RAG retrieval evaluation flow: comparing the retrieved document stack against a vector database baseline using binary and graded relevance rules.

Relevance Paradigms: Binary vs. Graded

Evaluation metrics are divided based on how relevance is labeled.

In the Binary Relevance paradigm, a document is either relevant (\(1\) or \(\checkmark\)) or irrelevant (\(0\) or \(\times\)).

This is the simplest way to annotate data. It is easy to construct and verify. Metrics using this model include Precision@K, Recall@K, Hit Rate@K, MRR, and MAP.

In the Graded Relevance paradigm, a document is given a score representing its level of utility.

For example, you can use a scale where \(3\) represents highly relevant, \(1\) represents partially relevant, and \(0\) represents irrelevant. This models search results more accurately. It supports nuanced differences in document utility. The primary metric using this model is NDCG.

Interactive Retrieval Simulator

Use this simulator to see how changes to the retrieval stack affect all six metrics. Adjust the evaluation depth \(K\) and total database relevant documents \(N\). Move documents to change their rank order. Toggle relevance in binary mode, or set specific utility scores in graded mode.

Retrieval Stack Simulator

Re-rank documents, toggle relevance, and adjust sliders to see changes instantly.

Real-Time Performance Metrics

Precision@K
0.60
3 / 5
Recall@K
0.50
3 / 6
Hit Rate@K
1.00
1
Reciprocal Rank (RR)
1.00
1 / 1
Average Precision (AP)
0.76
(1.0+0.67+0.6)/3
NDCG@K
0.96
DCG: 3.50 / IDCG: 3.63
How are these values calculated?

Here are the mathematical formulas for each metric:

Precision@K

Calculates the percentage of retrieved documents that are relevant up to depth \(K\).

Precision@K = (Relevant Docs in top K) / K
Recall@K

Measures the fraction of all relevant database documents \(N\) captured in the top \(K\). 

Recall@K = (Relevant Docs in top K) / N
Hit Rate@K

A simple check that returns \(1\) if at least one relevant document is found in the top \(K\), otherwise \(0\).

Hit Rate@K = 1 if (Relevant Docs in top K) > 0 else 0
Reciprocal Rank (RR)

The inverse rank of the first relevant document. If no relevant document is retrieved, the score is \(0\).

RR = 1 / rank_first
Average Precision (AP)

Evaluates list ranking quality by calculating the running average of precision at each relevant document position.

AP = (Sum_{i=1}^K Precision@i * rel_i) / (Relevant Docs in top K)
Normalized Discounted Cumulative Gain (NDCG@K)

Applies a logarithmic discount to relevance scores based on position, normalized by the ideal order score.

DCG@K = Sum_{i=1}^K rel_i / log_2(i + 1)
IDCG@K = Sum_{i=1}^K sorted_rel_i / log_2(i + 1)
NDCG@K = DCG@K / IDCG@K

Note: If you run Graded relevance mode, scores of 1, 2, and 3 are mapped to relevant (1) for binary metrics, while 0 maps to irrelevant (0).

Precision@K & Recall@K (The Binary Foundation)

These metrics measure the purity of your retrieved stack versus its coverage of the database.

Precision@K

Precision@K calculates the percentage of retrieved documents that are relevant.

For example, if you have three relevant documents in your top five:

\[\text{Precision@5} = \frac{\text{Relevant Documents in Top 5}}{5} = \frac{3}{5} = 0.60\ (60\%)\]

Recall@K

Recall@K measures the fraction of all relevant database documents (\(N\)) captured in the retrieved stack.

If the total number of relevant documents in the vector store is \(N=6\):

\[\text{Recall@5} = \frac{\text{Relevant Documents in Top 5}}{\text{Total Relevant in DB (N)}} = \frac{3}{6} = 0.50\ (50\%)\]

Order Insensitivity: Precision@K and Recall@K are order-insensitive within the top \(K\). They do not care if your best result is at Rank 1 or Rank 5. If you shuffle the stack positions, the final calculations remain exactly the same.

Hit Rate@K (Vector Search Benchmark)

Hit Rate is a simplified binary metric widely used in vector database performance benchmarks. It asks a binary question: Did you retrieve at least one relevant document in your top \(K\)?

To calculate the average Hit Rate over a query set \(Q\), you sum the query hits and divide by the total query count. For example, if you evaluate three queries with \(K=3\):

  • Query 1: \([\checkmark, \times, \times]\) → Hit Rate = 1
  • Query 2: \([\times, \times, \times]\) → Hit Rate = 0
  • Query 3: \([\times, \checkmark, \times]\) → Hit Rate = 1
\[\text{Average Hit Rate@3} = \frac{1 + 0 + 1}{3} \approx 0.67\ (67\%)\]

Hit Rate is useful for simple search tasks where any single correct document answers your user query.

Mean Reciprocal Rank (MRR)

MRR is an order-sensitive metric designed for navigational searches or single-answer lookups. It only evaluates the rank of the first relevant document. Any subsequent hits are ignored.

The Reciprocal Rank (RR) is the inverse of the first correct rank:

\[\text{RR} = \frac{1}{\text{rank}_{\text{first}}}\]

Consider two queries evaluated side-by-side:

  • Query A: The first relevant document appears at Rank 3. Subsequent hits are ignored. \[\text{RR}_A = \frac{1}{3} \approx 0.33\]
  • Query B: The first relevant document appears at Rank 1. \[\text{RR}_B = \frac{1}{1} = 1.00\]

Taking the mean across both queries yields:

\[\text{MRR} = \frac{\text{RR}_A + \text{RR}_B}{2} = \frac{0.33 + 1.00}{2} \approx 0.67\]

Mean Average Precision (MAP)

MAP is the standard metric for ranked lists using binary relevance. It evaluates the entire ranking quality by calculating the Precision at each relevant rank position.

First, calculate the Average Precision (AP) for a single query:

\[\text{AP} = \frac{\sum_{i=1}^{K} \text{Precision@i} \times rel_i}{\text{Total Retrieved Relevant Documents}}\]

For a query with relevant documents at ranks 1, 3, and 5:

  • At Rank 1 (Relevant): \(\text{P@1} = 1/1 = 1.00\)
  • At Rank 3 (Relevant): \(\text{P@3} = 2/3 \approx 0.67\)
  • At Rank 5 (Relevant): \(\text{P@5} = 3/5 = 0.60\)
\[\text{AP}_{\text{Good}} = \frac{1.00 + 0.67 + 0.60}{3} \approx 0.76\]

If the relevant items are pushed down to ranks 4 and 5:

  • At Rank 4 (Relevant): \(\text{P@4} = 1/4 = 0.25\)
  • At Rank 5 (Relevant): \(\text{P@5} = 2/5 = 0.40\)
\[\text{AP}_{\text{Bad}} = \frac{0.25 + 0.40}{2} \approx 0.33\]

To find MAP, you average the AP scores across your query set. MAP successfully penalizes queries that delay relevant results.

Normalized Discounted Cumulative Gain (NDCG)

NDCG is used when relevance is graded rather than binary. It relies on a logarithmic position discount. It penalizes highly relevant documents placed low in the stack.

To calculate NDCG, you must calculate Cumulative Gain (CG), Discounted Cumulative Gain (DCG), and Ideal DCG (IDCG).

NDCG Calculation Steps

Walk through the steps of calculating NDCG for a stack with utility scores \([3, 0, 1]\).

Step 1: Cumulative Gain (CG)

Sum the raw relevance scores without applying position discounts. For the stack \([3, 0, 1]\):

\[\text{CG} = \sum_{i=1}^{K} rel_i = 3 + 0 + 1 = 4\]

CG does not penalize ordering errors. If you shuffle the list to \([0, 1, 3]\), the CG remains 4.

Step 2: Discounted Cumulative Gain (DCG)

Apply a logarithmic discount to each score based on its rank index. This penalizes relevant documents placed lower in the list:

\[\text{DCG} = \sum_{i=1}^{K} \frac{rel_i}{\log_2(i + 1)}\]

For our list:

\[\text{DCG} = \frac{3}{\log_2(2)} + \frac{0}{\log_2(3)} + \frac{1}{\log_2(4)} = \frac{3}{1.0} + 0 + \frac{1}{2.0} = 3.5\]

Step 3: Ideal DCG (IDCG)

Determine the maximum possible score by sorting the retrieved documents in descending order. This produces the ideal ranking: \([3, 1, 0]\).

Calculate the DCG of this ideal list:

\[\text{IDCG} = \frac{3}{\log_2(2)} + \frac{1}{\log_2(3)} + \frac{0}{\log_2(4)} \approx 3.0 + \frac{1}{1.585} + 0 \approx 3.63\]

Step 4: Normalization (NDCG)

Normalize the DCG by dividing it by the IDCG. This scales the final metric value between \(0.0\) and \(1.0\):

\[\text{NDCG} = \frac{\text{DCG}}{\text{IDCG}} = \frac{3.5}{3.63} \approx 0.96\]

A perfect ranking scores 1.0. Lower scores indicate ordering issues.

Knowledge Check

Verify your understanding of these retrieval evaluation metrics.

You swap documents at Rank 2 and Rank 5 in a retrieval list. Which metrics will change?
Precision@5 and Recall@5
MAP and NDCG
Hit Rate@5 only
Correct! MAP and NDCG are order-sensitive. Toggling rank positions will shift their output. Precision, Recall, and Hit Rate are order-insensitive within the top \(K\).
Incorrect. Precision and Recall are order-insensitive inside the top \(K\). Think about which metrics evaluate positions.
Your RAG application provides a single definitive answer to factoid questions. Which metric should you prioritize?
Mean Reciprocal Rank (MRR)
Recall@K
Normalized Discounted Cumulative Gain (NDCG)
Correct! MRR is optimized for navigational or single-answer retrieval where you only care about the rank position of the first correct hit.
Incorrect. Recall measuring database coverage is too broad, and NDCG requires graded relevance. For single-answer lookups, the rank of the first hit is key.

Summary Selection Matrix

Use this table to choose the appropriate metric for your retrieval task.

Metric Relevance Type Order Sensitive? Best Use Case
Hit Rate / Recall Binary No Coverage: Confirms if the target text is present in the context window.
Precision Binary No Cost Control: Reduces noise in the prompt to minimize tokens and distractors.
MRR Binary Yes Factoid QA: Evaluates systems returning a single target answer.
MAP Binary Yes General Retrieval: Measures ranking quality across multiple relevant source documents.
NDCG Graded Yes Nuanced Search: Essential when documents contain degrees of relevance.