Build Reranking for RAG in Python

Build Reranking for RAG in Python

What You Will Build

In this tutorial, you will build reranking for RAG in Python to improve the quality of documents retrieved before they are sent to a large language model. Instead of trusting the initial retrieval order, your Retrieval-Augmented Generation (RAG) pipeline will use a cross-encoder model to evaluate each candidate document against the user’s query and reorder the results by relevance.

Reranking adds a second, more precise retrieval stage. BM25 and FAISS can quickly search a large knowledge base and return a set of candidate documents, while the reranker performs a deeper comparison on this smaller set to identify which chunks are actually the most relevant.

The resulting pipeline looks like this:

User Query
      │
      ▼
Hybrid Search
(BM25 + FAISS)
      │
      ▼
Candidate Documents
      │
      ▼
Cross-Encoder Reranker
      │
      ▼
Top Ranked Documents
      │
      ▼
Build Context
      │
      ▼
LLM
      │
      ▼
Final Answer

By the end of this tutorial, you will have a practical reranking pipeline that can sit on top of the hybrid search system built in the previous tutorial.

What You Will Learn

  • Why the initial retrieval ranking is not always accurate enough
  • How cross-encoder reranking works
  • How to retrieve a larger set of candidate documents
  • How to rerank candidates using their relevance to the user’s query
  • How to select only the highest-ranked chunks for the LLM
  • How to build reranking for RAG in Python as part of a production-oriented retrieval pipeline

This architecture separates fast candidate retrieval from precise relevance scoring. That distinction becomes increasingly important as your knowledge base grows and your RAG system needs to choose the best context from many potentially relevant documents.

Prerequisites

Before you build reranking for RAG in Python, make sure you already have a working retrieval pipeline. This tutorial extends the hybrid search system from the previous guide by adding a cross-encoder that evaluates and reorders the documents returned by the initial retrieval stage.

Your project should already include:

  • Python 3.10 or later
  • An OpenAI API key
  • Sentence Transformers
  • A FAISS vector index
  • BM25 keyword search
  • A collection of indexed document chunks
  • Metadata filtering (recommended)

Install the required libraries:

pip install openai sentence-transformers faiss-cpu rank-bm25 numpy

The sentence-transformers library provides cross-encoder models that can score a query and document together. Unlike embedding-based retrieval, where query and document vectors are generated independently, a cross-encoder analyzes both texts at the same time and produces a direct relevance score.

For this tutorial, we’ll use a lightweight pretrained cross-encoder:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder(
    "cross-encoder/ms-marco-MiniLM-L-6-v2"
)

You can find additional models and documentation in the official Sentence Transformers documentation.

To build reranking for RAG in Python efficiently, you should not run the cross-encoder against every document in your knowledge base. The initial retrieval stage first selects a relatively small group of candidates, and the more computationally expensive reranker then evaluates only those documents.

This creates an important two-stage architecture: fast retrieval first, precise reranking second. In the next step, we’ll examine why this additional ranking stage can improve the context ultimately sent to the language model.

Step 1 — Why RAG Needs Reranking

The first step to build reranking for RAG in Python is understanding why the initial retrieval results are not always ordered correctly. BM25, FAISS, and hybrid search are excellent at quickly finding potentially relevant documents, but their primary job is candidate retrieval rather than precise relevance evaluation.

Imagine that hybrid search retrieves 20 document chunks for the following question:

How can I improve retrieval accuracy in a RAG system?

Several chunks may discuss vector search, embeddings, BM25, chunking, metadata filtering, or retrieval optimization. All of them are related to the query, but some answer the question much more directly than others.

The initial ranking might look like this:

1. Introduction to vector embeddings
2. Improving retrieval with metadata
3. RAG retrieval optimization techniques
4. Choosing a FAISS index
5. Hybrid search with BM25
...

The third document may actually contain the best answer, even though the initial retriever ranked it below two less relevant chunks.

This happens because first-stage retrieval methods optimize for speed. FAISS compares vector representations, while BM25 evaluates lexical similarity. Neither method performs a deep query-document comparison using the full text of both inputs.

Reranking solves this problem by introducing a second retrieval stage:

Thousands of Documents
        │
        ▼
Fast Retrieval
        │
        ▼
Top 20 Candidates
        │
        ▼
Precise Reranking
        │
        ▼
Top 5 Documents
        │
        ▼
LLM Context

Instead of asking the reranker to evaluate thousands of documents, the retrieval system gives it only a small candidate set. The reranker can therefore use a more computationally expensive model to evaluate each candidate much more precisely.

This is the main reason to build reranking for RAG in Python. The first-stage retriever focuses on recall — finding potentially useful documents — while the reranker focuses on precision — moving the most relevant documents to the top.

In the next step, you’ll see how a cross-encoder performs this deeper query-document comparison and why it can produce more accurate relevance scores than embedding similarity alone.

Step 2 — Understand Cross-Encoder Reranking

To build reranking for RAG in Python, you need to understand how a cross-encoder differs from the embedding model used during the initial retrieval stage. Both approaches estimate relevance, but they process the query and document in fundamentally different ways.

With embedding-based retrieval, the query and each document are encoded separately:

Query
  ↓
Embedding Model
  ↓
Query Vector
  ↓
Vector Similarity
  ↑
Document Vector
  ↑
Embedding Model
  ↑
Document

This architecture is extremely efficient because document embeddings can be calculated once and stored in FAISS. At search time, only the query embedding needs to be generated before comparing it with thousands or millions of stored vectors.

A cross-encoder works differently. Instead of generating independent vectors, it processes the query and document together:

Query + Document
       │
       ▼
  Cross-Encoder
       │
       ▼
Relevance Score

Because the model sees both texts simultaneously, it can analyze relationships between individual words, phrases, and concepts in the query and document. This usually provides a more precise relevance estimate than comparing independently generated embeddings.

For example, create the cross-encoder:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder(
    "cross-encoder/ms-marco-MiniLM-L-6-v2"
)

Then create query-document pairs:

pairs = [
    [query, document["text"]]
    for document in candidate_documents
]

The model can score all candidate pairs in a single call:

scores = reranker.predict(pairs)

Each returned value represents the model’s estimate of how relevant a particular document is to the query. You can attach these scores to the candidate documents and sort them from highest to lowest relevance.

for document, score in zip(candidate_documents, scores):
    document["rerank_score"] = float(score)

ranked_documents = sorted(
    candidate_documents,
    key=lambda x: x["rerank_score"],
    reverse=True
)

The trade-off is computational cost. A bi-encoder can precompute document vectors and search them efficiently, while a cross-encoder must process every query-document pair at request time. Running it across an entire knowledge base would therefore be unnecessarily expensive.

This is why a production RAG pipeline typically uses both approaches. Fast retrieval generates a relatively small candidate set, and the cross-encoder performs precise relevance scoring only on those candidates. When you build reranking for RAG in Python this way, you combine the scalability of vector and keyword retrieval with the higher precision of a cross-encoder.

In the next step, you’ll retrieve a larger candidate set from the hybrid search pipeline and prepare those documents for reranking.

Step 3 — Retrieve Candidate Documents

Before reranking can begin, your RAG system needs to retrieve a set of candidate documents. The goal at this stage is not to find only the final five chunks, but to collect a broader group of potentially relevant documents that the cross-encoder can evaluate more precisely.

Suppose your hybrid search pipeline normally returns the top 5 documents:

results = hybrid_search(
    query,
    top_k=5
)

When adding reranking, retrieve a larger candidate set instead:

candidate_documents = hybrid_search(
    query,
    top_k=20
)

These 20 documents become the input for the reranking stage. The idea is simple: the initial retriever should cast a wider net, while the reranker decides which candidates deserve the highest positions.

For example, the initial hybrid search might return:

1. Vector embeddings explained
2. Metadata filtering for RAG
3. Improving retrieval accuracy
4. FAISS index optimization
5. Cross-encoder reranking
6. BM25 keyword search
7. RAG evaluation techniques
...
20. Document chunking strategies

At this point, you should avoid treating the original ranking as the final result. The purpose of candidate retrieval is primarily to ensure that the truly relevant documents are present somewhere in the candidate set.

This introduces an important distinction between the two stages:

Stage 1: Retrieval

Large Knowledge Base
        ↓
Top 20 Candidates

Goal: Find potentially relevant documents


Stage 2: Reranking

Top 20 Candidates
        ↓
Top 5 Documents

Goal: Identify the most relevant documents

Choosing the candidate count involves a trade-off. If you retrieve too few documents, the best chunk may never reach the reranker. If you retrieve too many, the cross-encoder must perform more query-document comparisons, increasing latency and computational cost.

A practical starting point is to retrieve around 20 candidates and rerank them down to the best 5. These numbers are not universal and should eventually be tuned using retrieval evaluation on your own dataset.

RETRIEVAL_TOP_K = 20
RERANK_TOP_K = 5

candidate_documents = hybrid_search(
    query,
    top_k=RETRIEVAL_TOP_K
)

When you build reranking for RAG in Python, this separation between candidate retrieval and final selection is essential. BM25 and FAISS search broadly and efficiently, while the cross-encoder receives a manageable number of candidates for more accurate relevance scoring.

In the next step, you’ll pass these candidate documents to the cross-encoder, calculate relevance scores, and reorder the results before building the final context for the language model.

Step 4 — Rerank Documents with a Cross-Encoder

Now that the retrieval pipeline has produced a set of candidate documents, the next step is to evaluate each candidate with the cross-encoder. This is the core stage when you build reranking for RAG in Python because the original retrieval order is replaced with a more precise relevance ranking.

Start with the candidate documents returned by hybrid search:

candidate_documents = hybrid_search(
    query,
    top_k=20
)

The cross-encoder expects pairs containing the user query and one candidate document. Create these pairs from the retrieved chunks:

pairs = [
    (query, document["text"])
    for document in candidate_documents
]

Now pass all query-document pairs to the reranking model:

scores = reranker.predict(pairs)

The model returns one relevance score for each candidate. Attach those scores to the corresponding documents:

for document, score in zip(
    candidate_documents,
    scores
):
    document["rerank_score"] = float(score)

Next, sort the documents by their new relevance scores:

ranked_documents = sorted(
    candidate_documents,
    key=lambda x: x["rerank_score"],
    reverse=True
)

Finally, keep only the highest-ranked documents:

top_documents = ranked_documents[:5]

The difference between the original and reranked results might look like this:

Initial Retrieval          After Reranking

1. Vector Embeddings       1. RAG Retrieval Optimization
2. Metadata Filtering      2. Cross-Encoder Reranking
3. RAG Optimization        3. Hybrid Search for RAG
4. FAISS Indexes           4. Metadata Filtering
5. Cross-Encoder           5. Vector Embeddings

Notice that reranking does not discover new documents. It works only with the candidates provided by the first retrieval stage. If an important document is missing from the candidate set, the cross-encoder cannot recover it. This is why good first-stage retrieval remains essential.

You can wrap the entire reranking process in a reusable function:

def rerank_documents(
    query,
    documents,
    top_k=5
):
    pairs = [
        (query, doc["text"])
        for doc in documents
    ]

    scores = reranker.predict(pairs)

    for doc, score in zip(documents, scores):
        doc["rerank_score"] = float(score)

    ranked = sorted(
        documents,
        key=lambda x: x["rerank_score"],
        reverse=True
    )

    return ranked[:top_k]

With this function, your retrieval pipeline can search broadly using BM25 and FAISS and then apply a more precise relevance model before sending context to the LLM. This two-stage approach is the central principle behind reranking for Retrieval-Augmented Generation.

In the next step, you’ll connect retrieval, reranking, context construction, and answer generation into one complete RAG pipeline.

Step 5 — Build a Reranking RAG Pipeline

Now it’s time to combine the individual components and build reranking for RAG in Python as part of a complete retrieval pipeline. The system will first retrieve a broad set of candidate documents, rerank them with the cross-encoder, select the best chunks, and only then send the resulting context to the language model.

The complete workflow looks like this:

User Question
      │
      ▼
Metadata Filtering
      │
      ▼
Hybrid Search
(BM25 + FAISS)
      │
      ▼
Top 20 Candidates
      │
      ▼
Cross-Encoder Reranking
      │
      ▼
Top 5 Documents
      │
      ▼
Build Context
      │
      ▼
LLM
      │
      ▼
Final Answer

Start by retrieving more documents than you ultimately intend to send to the language model:

candidate_documents = hybrid_search(
    question,
    top_k=20
)

Next, pass those candidates through the reranking function created in the previous step:

top_documents = rerank_documents(
    question,
    candidate_documents,
    top_k=5
)

Build the final context from only the highest-ranked chunks:

context = "\n\n".join(
    document["text"]
    for document in top_documents
)

The complete RAG function can now be structured like this:

def ask_rag(question):

    candidate_documents = hybrid_search(
        question,
        top_k=20
    )

    top_documents = rerank_documents(
        question,
        candidate_documents,
        top_k=5
    )

    context = "\n\n".join(
        doc["text"]
        for doc in top_documents
    )

    messages = [
        {
            "role": "system",
            "content": (
                "Answer the question using only "
                "the provided context."
            )
        },
        {
            "role": "user",
            "content": f"""
Context:

{context}

Question:

{question}
"""
        }
    ]

    response = client.chat.completions.create(
        model="gpt-4.1-mini",
        messages=messages
    )

    return response.choices[0].message.content

The important point is that the LLM remains the final generation component rather than the retrieval engine. Document selection happens before generation: hybrid search finds candidates, the cross-encoder improves their ordering, and only the highest-ranked chunks become part of the prompt.

When you build reranking for RAG in Python using this architecture, each component has a clearly defined responsibility. Fast retrieval provides broad coverage, reranking improves precision, and the language model generates an answer from a smaller and more relevant context.

This architecture also provides a useful optimization opportunity. Instead of sending 20 retrieved chunks to the LLM, the system may send only the best 5 after reranking. This can reduce unnecessary context, lower token consumption, and prevent weaker documents from competing with highly relevant information inside the prompt.

Your RAG pipeline now contains the major retrieval stages used in many production systems: metadata filtering, lexical retrieval, semantic retrieval, hybrid ranking, and cross-encoder reranking.

In the next section, you’ll test the pipeline and compare the document order before and after reranking to determine whether the additional retrieval stage actually improves relevance.

Test Reranking

After you build reranking for RAG in Python, you should verify that the additional ranking stage actually improves retrieval quality. The simplest test is to compare the order of documents returned by hybrid search with the order produced by the cross-encoder.

Consider the following query:

How can I improve retrieval accuracy in a RAG system?

Hybrid search might return these five candidates:

1. Vector Embeddings Explained
2. FAISS Similarity Search
3. Improving Retrieval Accuracy in RAG
4. Metadata Filtering
5. Cross-Encoder Reranking

All five documents are related to the query, so the initial retrieval stage has done its job. However, the third and fifth documents may provide much more direct answers than the first two.

After cross-encoder reranking, the order might change:

1. Improving Retrieval Accuracy in RAG     8.74
2. Cross-Encoder Reranking                 7.91
3. Metadata Filtering                      5.63
4. FAISS Similarity Search                 4.82
5. Vector Embeddings Explained             3.95

You can inspect the results directly in Python:

for rank, document in enumerate(
    top_documents,
    start=1
):
    print(
        rank,
        document["rerank_score"],
        document["text"][:100]
    )

Do not evaluate reranking only by checking whether the scores changed. The important question is whether the documents that best answer the user’s query move toward the top of the result list.

A useful test set should contain different types of queries, including natural-language questions, exact technical terminology, product identifiers, abbreviations, and queries where several documents discuss closely related topics.

For a more systematic evaluation, record the relevant documents for a collection of test queries and compare retrieval metrics before and after reranking. Metrics such as Precision@K, Recall@K, and Mean Reciprocal Rank can help determine whether the reranker consistently improves the ordering rather than merely changing it.

This evaluation step is important when you build reranking for RAG in Python because reranking introduces additional latency and computation. The extra stage is valuable only if the improvement in retrieval quality justifies that cost for your application.

Once testing confirms that the cross-encoder consistently moves better documents toward the top, you have a much stronger retrieval pipeline for supplying relevant context to the language model.

Where to Go Next

Congratulations! You have learned how to build reranking for RAG in Python and add a second relevance stage to your Retrieval-Augmented Generation pipeline. Your system can now retrieve candidates quickly with hybrid search and then use a cross-encoder to select the documents that best match the user’s question.

At this point, your retrieval architecture has evolved considerably:

Metadata Filtering
        ↓
Hybrid Search
(BM25 + FAISS)
        ↓
Candidate Documents
        ↓
Cross-Encoder Reranking
        ↓
Top Documents
        ↓
LLM
        ↓
Answer

You can continue developing the system with the previous tutorials in this practical RAG series:

The next major challenge is no longer adding another retrieval method. It is determining whether the complete RAG system actually performs well. Individual examples can look convincing while hiding retrieval failures, weak answers, or regressions introduced by later changes.

The next tutorial will therefore focus on evaluating a RAG system in Python. You’ll create a test dataset, measure retrieval quality, evaluate generated answers, and compare different versions of the pipeline using repeatable metrics.

Frequently Asked Questions

Why should I build reranking for RAG in Python?

Reranking improves the order of documents returned by the initial retrieval stage. BM25 and FAISS are optimized for fast candidate retrieval, while a cross-encoder performs a deeper query-document comparison. This helps ensure that the most relevant chunks are placed at the top before context is sent to the language model.

What is the difference between retrieval and reranking?

Retrieval searches a large knowledge base and quickly selects potentially relevant documents. Reranking works on this smaller candidate set and evaluates each document more precisely. In a typical RAG pipeline, retrieval may select 20 candidates and the reranker may reduce them to the best 5.

Why use a cross-encoder for RAG reranking?

A cross-encoder processes the query and document together instead of creating independent embeddings. This allows the model to examine relationships between words and concepts directly, often producing more accurate relevance scores than vector similarity alone.

How many documents should I rerank?

There is no universal number. A practical starting point is to retrieve approximately 20 candidates and rerank them down to the best 5. Larger candidate sets may improve recall but also increase latency and computational cost, so the values should be tested on your own dataset.

Can reranking work with hybrid search?

Yes. Hybrid search and reranking complement each other well. BM25 and FAISS first provide broad lexical and semantic retrieval, while the cross-encoder then reorders those candidates according to their relevance to the user’s query.

Does reranking reduce RAG hallucinations?

Reranking can help reduce hallucinations indirectly by providing the language model with more relevant context. However, it does not guarantee factual answers. Prompt design, document quality, retrieval evaluation, and answer validation remain important parts of a reliable Retrieval-Augmented Generation system.

Conclusion

In this tutorial, you learned how to build reranking for RAG in Python using a cross-encoder model. Instead of relying entirely on the ranking produced by BM25 or FAISS, your retrieval pipeline now performs a second, more precise relevance evaluation before sending documents to the language model.

You built a complete two-stage retrieval architecture: fast hybrid search retrieves a broad candidate set, and the cross-encoder reranks those candidates to identify the documents that best match the user’s question.

You also learned an important principle of production RAG systems: retrieval and reranking solve different problems. The initial retriever should provide strong recall and efficiently find potentially useful documents, while the reranker improves precision by selecting the best results from that candidate set.

The complete pipeline now looks like this:

User Query
      ↓
Metadata Filtering
      ↓
BM25 + FAISS
      ↓
Hybrid Retrieval
      ↓
Candidate Documents
      ↓
Cross-Encoder Reranking
      ↓
Top Documents
      ↓
LLM
      ↓
Final Answer

When you build reranking for RAG in Python using this architecture, you gain much more control over which information reaches the language model. This can improve retrieval relevance, reduce unnecessary context, and provide a stronger foundation for accurate RAG responses.

The next step is to stop improving the pipeline based only on individual examples and start measuring it systematically. In the next tutorial, you’ll learn how to evaluate a RAG system in Python using retrieval and answer-quality metrics so you can determine whether each change actually improves performance.

Scroll to Top