
Traditional RAG systems typically rely on vector search to retrieve relevant information. While semantic search works well for understanding meaning, it can struggle with exact keywords, product identifiers, technical terms, and specific document references.
Hybrid search for RAG addresses this limitation by combining two complementary retrieval methods: BM25 keyword search and FAISS vector search. BM25 identifies documents containing relevant terms, while FAISS retrieves passages based on semantic similarity.
In this tutorial, we will build a hybrid search system in Python using BM25, FAISS, Sentence Transformers, and Reciprocal Rank Fusion (RRF). We will implement both retrieval methods, combine their rankings, and create a reusable hybrid retriever for a RAG pipeline.
By the end, you will understand how hybrid retrieval works, when it improves search relevance, and how to integrate it into an existing document indexing architecture.
What Is Hybrid Search in RAG?
Hybrid search for RAG is a retrieval technique that combines keyword-based search with semantic vector search. Instead of relying on a single ranking algorithm, it uses multiple retrieval methods to identify relevant document chunks.
The two approaches complement each other. Keyword search finds exact terms and technical identifiers, while semantic search identifies passages with similar meaning, even when they use different words.
How Hybrid Retrieval Works
When a user submits a query, the system processes it through two independent search engines:
- BM25: ranks document chunks based on query terms, term frequency, and document length.
- FAISS: searches vector embeddings to find semantically similar chunks.
- Rank fusion: combines both result lists into a unified ranking.
- RAG generation: passes the highest-ranked chunks to the language model as context.
For example, consider the query “How to configure the FX3U PLC communication port?”
BM25 can prioritize documents containing the exact identifier FX3U, while semantic search may retrieve explanations about PLC serial communication, RS485 settings, and Modbus RTU configuration.
Combining these results increases the chances of retrieving both the exact technical reference and useful contextual information.
Why Use Reciprocal Rank Fusion?
BM25 and vector search produce scores on different scales. Directly adding these scores can distort the final ranking.
Reciprocal Rank Fusion (RRF) avoids this problem by combining document positions rather than raw similarity scores. Documents ranked highly by either retrieval method receive higher combined scores.
Hybrid search does not guarantee better results for every query, but it is particularly useful for technical documentation, product catalogs, enterprise knowledge bases, and collections containing specialized terminology.
BM25 vs. Vector Search: Key Differences
BM25 and vector search solve different retrieval problems. Understanding their strengths and limitations helps explain why combining them can improve hybrid search for RAG.
BM25: Keyword-Based Retrieval
BM25 is a statistical ranking algorithm that evaluates how well document terms match a search query. It considers term frequency, document length, and how common each term is across the collection.
BM25 is particularly effective when queries contain exact product names, error codes, model numbers, or technical terminology. However, it may miss relevant passages that express the same idea using different words.
Vector Search: Semantic Retrieval
Vector search converts text into numerical embeddings that represent semantic meaning. FAISS efficiently searches these vectors to identify similar passages.
For example, a query about reducing database response time may retrieve documents discussing query optimization, even without matching keywords.
However, embedding models do not always distinguish similar-looking technical identifiers or preserve the importance of exact terminology.
For a deeper understanding of vector similarity search, see the official FAISS documentation , which explains the library’s indexing and nearest-neighbor search capabilities.
BM25 vs. FAISS Comparison
| Feature | BM25 | FAISS Vector Search |
|---|---|---|
| Search principle | Lexical relevance | Semantic similarity |
| Exact identifiers | Generally strong | Model-dependent |
| Synonyms and paraphrases | Limited | Generally strong |
| Embedding generation | Not required | Required |
| Ranking | BM25 relevance score | Vector similarity or distance |
Neither approach is universally superior. BM25 provides precise lexical matching, while vector search captures relationships between concepts.
A hybrid retriever combines their complementary results, making it especially useful when a RAG knowledge base contains both natural-language explanations and specialized technical information.
Prerequisites and Project Setup
Before implementing hybrid search for RAG, we need a Python environment with libraries for keyword retrieval, text embeddings, and vector similarity search.
Install the Required Libraries
Use Python 3.11 or newer. Create a virtual environment and install the dependencies:
python -m venv .venv
pip install rank-bm25 faiss-cpu sentence-transformers numpy
Each library has a specific role:
- rank-bm25: implements BM25 keyword ranking.
- faiss-cpu: provides efficient vector similarity search.
- sentence-transformers: generates semantic text embeddings.
- NumPy: handles numerical arrays and vector data.
Project Structure
We will organize the implementation into separate modules:
hybrid-rag/
├── main.py
├── documents.py
├── bm25_search.py
├── vector_search.py
├── fusion.py
├── hybrid_retriever.py
└── requirements.txt
The example uses a small in-memory document collection to demonstrate the retrieval algorithms without introducing database configuration or background workers.
Both search engines will use the same stable chunk_id values. This allows us to merge their results, remove duplicates, and retrieve the original text after ranking.
For production systems, the same retrieval components can be connected to a persistent document registry and an existing incremental indexing pipeline.
Step 1 — Prepare Documents and Embeddings
The first step in building hybrid search for RAG is preparing a document collection that both retrieval engines can search.
For this tutorial, we will use a small collection of technical passages about PLC controllers, industrial communication, and database optimization.
Create a Document Collection
Create a file named documents.py:
DOCUMENTS = [
{
"chunk_id": 1,
"text": (
"The FX3U PLC supports serial communication "
"using RS485 and Modbus RTU protocols."
)
},
{
"chunk_id": 2,
"text": (
"Configure the serial port baud rate, "
"parity, and slave address before "
"connecting industrial controllers."
)
},
{
"chunk_id": 3,
"text": (
"Modbus RTU enables communication between "
"programmable logic controllers and "
"industrial monitoring devices."
)
},
{
"chunk_id": 4,
"text": (
"Database query optimization reduces "
"response time through indexing and "
"efficient execution plans."
)
},
{
"chunk_id": 5,
"text": (
"Vector databases retrieve semantically "
"similar documents using numerical "
"text embeddings."
)
}
]
Each document contains a stable identifier and a text passage. In a production RAG pipeline, these records would typically represent chunks extracted from PDF, HTML, or other source documents.
Generate Text Embeddings
Next, create vector_search.py and initialize a Sentence Transformers model:
import numpy as np
from sentence_transformers import SentenceTransformer
from documents import DOCUMENTS
model = SentenceTransformer(
"sentence-transformers/all-MiniLM-L6-v2"
)
texts = [
document["text"]
for document in DOCUMENTS
]
embeddings = model.encode(
texts,
normalize_embeddings=True
)
embeddings = np.asarray(
embeddings,
dtype=np.float32
)
print("Embedding shape:", embeddings.shape)
The selected model generates 384-dimensional embeddings. With five documents, the resulting array has the shape (5, 384).
We normalize embeddings so that their inner product corresponds to cosine similarity. This will allow FAISS to rank documents by semantic similarity.
Preserve Document Identifiers
The position of an embedding in the NumPy array is not necessarily the document’s permanent identifier.
We will therefore maintain an explicit mapping between FAISS vector IDs and document chunk_id values. BM25 will use the same identifiers when returning its ranked results.
This shared identity is essential for combining retrieval results without confusing documents or introducing duplicate passages.
Step 2 — Implement BM25 Keyword Search
BM25 ranks documents by matching query terms against the indexed text. Unlike semantic search, it does not require embeddings or a neural network.
In our hybrid search for RAG pipeline, BM25 will handle exact terminology, technical identifiers, and keyword-based retrieval.
Build the BM25 Index
Create bm25_search.py:
import re
import numpy as np
from rank_bm25 import BM25Okapi
from documents import DOCUMENTS
def tokenize(text: str) -> list[str]:
return re.findall(
r"[a-z0-9]+",
text.lower()
)
class BM25Search:
def __init__(self, documents=DOCUMENTS):
self.documents = documents
self.tokenized_corpus = [
tokenize(doc["text"])
for doc in documents
]
self.index = BM25Okapi(
self.tokenized_corpus
)
def search(self, query: str, top_k=5):
tokens = tokenize(query)
if not tokens:
return []
scores = self.index.get_scores(tokens)
ranked_indices = np.argsort(
-scores,
kind="stable"
)
results = []
for index in ranked_indices:
score = float(scores[index])
if score <= 0:
continue
document = self.documents[index]
results.append({
"chunk_id": document["chunk_id"],
"text": document["text"],
"score": score
})
if len(results) == top_k:
break
return results
The tokenizer converts text to lowercase and extracts alphanumeric terms. This preserves identifiers such as FX3U and RS485, although more advanced tokenization may be needed for identifiers containing hyphens or other special characters.
Test Keyword Retrieval
Create a temporary test in main.py:
from bm25_search import BM25Search
search_engine = BM25Search()
results = search_engine.search(
"FX3U PLC RS485",
top_k=3
)
for result in results:
print(
result["chunk_id"],
round(result["score"], 3),
result["text"]
)
The passage containing the exact FX3U and RS485 identifiers should rank highly because these terms are relatively distinctive within our collection.
Notice that the method returns both a relevance score and the original chunk_id. Later, we will use these identifiers to combine BM25 results with FAISS semantic search.
Limitations of BM25
BM25 depends on lexical overlap. A query about industrial device communication may not retrieve every relevant passage about serial protocols if the wording differs significantly.
This is where semantic retrieval becomes useful. By combining both methods, we can preserve precise keyword matching while also discovering conceptually related information.
Step 3 — Implement FAISS Semantic Search
Unlike BM25, semantic search retrieves documents based on meaning rather than exact keyword matches. This allows a RAG system to find relevant passages even when the query uses different terminology.
We will use FAISS to build the semantic component of our hybrid search for RAG pipeline.
Create the FAISS Vector Index
Replace the contents of vector_search.py with the following implementation:
import faiss
import numpy as np
from sentence_transformers import SentenceTransformer
from documents import DOCUMENTS
class VectorSearch:
def __init__(self, documents=DOCUMENTS):
self.documents = documents
self.document_map = {
doc["chunk_id"]: doc
for doc in documents
}
self.model = SentenceTransformer(
"sentence-transformers/all-MiniLM-L6-v2"
)
texts = [
doc["text"]
for doc in documents
]
embeddings = self.model.encode(
texts,
normalize_embeddings=True
)
vectors = np.asarray(
embeddings,
dtype=np.float32
)
dimension = vectors.shape[1]
self.index = faiss.IndexIDMap2(
faiss.IndexFlatIP(dimension)
)
vector_ids = np.array(
[doc["chunk_id"] for doc in documents],
dtype=np.int64
)
self.index.add_with_ids(
vectors,
vector_ids
)
def search(self, query: str, top_k=5):
if not query.strip() or top_k <= 0:
return []
query_vector = self.model.encode(
[query],
normalize_embeddings=True
)
query_vector = np.asarray(
query_vector,
dtype=np.float32
)
scores, ids = self.index.search(
query_vector,
min(top_k, self.index.ntotal)
)
results = []
for score, chunk_id in zip(
scores[0], ids[0]
):
if chunk_id == -1:
continue
document = self.document_map[
int(chunk_id)
]
results.append({
"chunk_id": int(chunk_id),
"text": document["text"],
"score": float(score)
})
return results
The IndexFlatIP index performs exact nearest-neighbor search using inner product similarity. Because our embeddings are normalized, these scores correspond to cosine similarity.
The IndexIDMap2 wrapper associates each vector with its original chunk_id. This eliminates the need to infer document identity from vector positions.
Test Semantic Retrieval
Update main.py:
from vector_search import VectorSearch
search_engine = VectorSearch()
results = search_engine.search(
"How do industrial controllers exchange data?",
top_k=3
)
for result in results:
print(
result["chunk_id"],
round(result["score"], 3),
result["text"]
)
The search should prioritize passages discussing PLC communication, serial connections, and industrial protocols, even though the query does not explicitly mention FX3U or Modbus RTU.
Why Stable Vector IDs Matter
Both BM25 and FAISS now return results with the same chunk_id values. This is essential because the same document may appear in both result lists.
Using stable identifiers allows us to recognize matching documents, combine their rankings, and avoid duplicate context passages.
For larger document collections, FAISS also supports approximate nearest-neighbor indexes that trade some retrieval accuracy for improved search performance. Our small example uses exact search to keep the implementation straightforward.
Step 4 — Combine Results Using Reciprocal Rank Fusion (RRF)
Our BM25 and FAISS search engines produce two independent ranked lists. The next step in building hybrid search for RAG is combining these results into a single ranking.
A simple addition of BM25 and cosine similarity scores would be unreliable because the two algorithms use different scoring scales.
How Reciprocal Rank Fusion Works
Reciprocal Rank Fusion (RRF) assigns a score to each document based on its position in the search results:
RRF(d) = Σ 1 / (k + rank(d))
Here, rank(d) is the document’s position in a result list, starting from 1. The constant k controls how strongly the algorithm favors top-ranked results. A commonly used value is 60.
A document appearing in both result lists receives contributions from both methods. Documents appearing in only one list can still be included in the final ranking.
Implement RRF in Python
Create fusion.py:
from collections import defaultdict
def reciprocal_rank_fusion(
result_lists: list[list[dict]],
k: int = 60,
top_k: int = 5
) -> list[dict]:
if k < 0:
raise ValueError("k must be non-negative")
if top_k <= 0:
return []
scores = defaultdict(float)
documents = {}
for results in result_lists:
seen = set()
for rank, result in enumerate(
results, start=1
):
chunk_id = result["chunk_id"]
if chunk_id in seen:
continue
seen.add(chunk_id)
scores[chunk_id] += 1.0 / (k + rank)
documents[chunk_id] = result["text"]
ranked = sorted(
scores,
key=lambda chunk_id: (
-scores[chunk_id],
chunk_id
)
)
return [
{
"chunk_id": chunk_id,
"text": documents[chunk_id],
"score": scores[chunk_id]
}
for chunk_id in ranked[:top_k]
]
The function accepts any number of ranked result lists. It calculates a combined score for each unique chunk_id and sorts documents by their final RRF scores.
The seen set prevents duplicate entries within one search engine’s results from contributing multiple times.
Test Rank Fusion
Consider two simplified result lists:
from fusion import reciprocal_rank_fusion
bm25_results = [
{"chunk_id": 1, "text": "PLC manual"},
{"chunk_id": 2, "text": "Serial configuration"},
{"chunk_id": 3, "text": "Modbus communication"}
]
vector_results = [
{"chunk_id": 3, "text": "Modbus communication"},
{"chunk_id": 1, "text": "PLC manual"},
{"chunk_id": 5, "text": "Vector database"}
]
results = reciprocal_rank_fusion(
[bm25_results, vector_results],
k=60,
top_k=3
)
for result in results:
print(
result["chunk_id"],
round(result["score"], 5)
)
The expected ranking is:
1 0.03252
3 0.03227
2 0.01613
Document 1 ranks first because it appears in both result lists at strong positions. Document 3 also benefits from appearing in both searches.
Choosing the RRF Parameter
A smaller k places more emphasis on differences between the highest-ranked positions. A larger value reduces those differences and makes contributions from lower-ranked results more similar.
The default value of 60 is a practical starting point, not a universally optimal setting. Test different values against representative queries and relevance judgments before choosing one for production.
RRF provides a straightforward way to combine lexical and semantic retrieval without calibrating their individual relevance scores.
Step 5 — Build a Hybrid Retriever in Python
We now have three independent components: BM25 keyword search, FAISS semantic search, and Reciprocal Rank Fusion. The final step is to combine them into a reusable hybrid search for RAG interface.
Create the Hybrid Retriever
Create hybrid_retriever.py:
from documents import DOCUMENTS
from bm25_search import BM25Search
from vector_search import VectorSearch
from fusion import reciprocal_rank_fusion
class HybridRetriever:
def __init__(self, documents=DOCUMENTS):
self.bm25 = BM25Search(documents)
self.vector = VectorSearch(documents)
def search(
self,
query: str,
top_k: int = 5,
candidate_k: int = 10
) -> list[dict]:
if not query.strip() or top_k <= 0:
return []
candidate_k = max(candidate_k, top_k)
bm25_results = self.bm25.search(
query,
top_k=candidate_k
)
vector_results = self.vector.search(
query,
top_k=candidate_k
)
return reciprocal_rank_fusion(
[bm25_results, vector_results],
k=60,
top_k=top_k
)
The candidate_k parameter determines how many results each search engine contributes to rank fusion. The top_k parameter controls how many documents the hybrid retriever returns.
Retrieving more candidates than the final result count gives RRF a larger pool of potentially relevant passages.
Run a Hybrid Search Query
Update main.py:
from hybrid_retriever import HybridRetriever
retriever = HybridRetriever()
query = "How to configure FX3U communication?"
results = retriever.search(
query,
top_k=3,
candidate_k=5
)
for rank, result in enumerate(results, start=1):
print(
f"{rank}. Chunk {result['chunk_id']}"
)
print(result["text"])
print(
f"RRF score: {result['score']:.5f}"
)
print()
The retriever searches for exact identifiers such as FX3U while also considering semantically related passages about serial communication and industrial controllers.
Connect the Retriever to a RAG Pipeline
The returned passages can be assembled into context for a language model:
context = "\n\n".join(
result["text"]
for result in results
)
prompt = f"""Answer the question using only
the provided context. If the context is
insufficient, say so.
Context:
{context}
Question:
{query}
Answer:"""
The prompt can then be passed to the language model used by your RAG application.
For production use, retain document identifiers and source metadata alongside the retrieved text so the generated answer can reference its supporting documents.
Our hybrid retriever is now ready to use. The next step is to measure whether combining BM25 and FAISS actually improves retrieval quality.
Test and Evaluate Hybrid Search
Building hybrid search for RAG is only the first step. We also need to measure whether combining BM25 and FAISS improves retrieval quality compared with using either method independently.
Prepare Evaluation Queries
Create evaluate.py with a small set of queries and manually identified relevant chunks:
from hybrid_retriever import HybridRetriever
TEST_QUERIES = [
{
"query": "FX3U PLC RS485",
"relevant_ids": {1}
},
{
"query": "industrial device communication",
"relevant_ids": {1, 2, 3}
},
{
"query": "improve database response time",
"relevant_ids": {4}
},
{
"query": "semantic document retrieval",
"relevant_ids": {5}
}
]
The relevant_ids field contains the chunk identifiers considered relevant to each query. These labels form a small evaluation dataset.
Calculate Recall@K
Recall@K measures the proportion of relevant documents retrieved among the top K results.
def recall_at_k(results, relevant_ids):
retrieved_ids = {
item["chunk_id"]
for item in results
}
if not relevant_ids:
return 0.0
return len(
retrieved_ids & relevant_ids
) / len(relevant_ids)
A Recall@3 of 1.0 means all labeled relevant chunks were found within the first three results. It does not necessarily mean that every retrieved chunk was relevant.
Compare All Three Retrieval Methods
Add the evaluation loop:
retriever = HybridRetriever()
methods = {
"BM25": retriever.bm25.search,
"FAISS": retriever.vector.search,
"Hybrid": retriever.search
}
for name, search in methods.items():
scores = []
for test in TEST_QUERIES:
results = search(
test["query"],
top_k=3
)
scores.append(
recall_at_k(
results,
test["relevant_ids"]
)
)
average = sum(scores) / len(scores)
print(f"{name}: Recall@3 = {average:.3f}")
Run the evaluation:
python evaluate.py
The script prints the average Recall@3 for each retrieval method. Actual values depend on the embedding model, document collection, and selected queries.
Interpret the Results
Compare the methods across different query types. BM25 may perform particularly well on exact identifiers, while FAISS may retrieve more relevant passages for paraphrased questions.
Hybrid retrieval is useful when it improves results across the query types that matter to your application. If its performance is worse, investigate tokenization, candidate counts, embedding quality, and RRF parameters.
This five-document example is a functional test, not a statistically meaningful benchmark. For production RAG, evaluate a larger collection with representative queries, multiple relevance judgments, and additional metrics such as MRR and nDCG.
Production Considerations for Hybrid Search
Our Python implementation demonstrates how to combine BM25 and FAISS, but production hybrid search for RAG requires additional attention to scalability, index updates, retrieval latency, and data consistency.
Keep BM25 and FAISS Indexes Synchronized
Both retrieval engines must reference the same document collection and use stable chunk identifiers.
When documents are added, modified, or deleted, update both indexes. Otherwise, the hybrid retriever may return outdated passages or inconsistent results.
For reliable updates, use versioned index snapshots or a coordinated indexing process. The document synchronization pipeline can trigger these updates automatically.
Scale Keyword and Vector Search
Our example uses rank-bm25, which keeps the tokenized corpus in memory and evaluates documents during search. This is suitable for small collections but may become inefficient as the dataset grows.
For larger workloads, consider a search engine with an inverted index, such as OpenSearch or Elasticsearch.
FAISS offers approximate nearest-neighbor indexes for larger vector collections. These can reduce search latency at the cost of additional configuration and potentially lower retrieval recall.
Optimize Retrieval Performance
BM25 and FAISS searches are independent and can run concurrently. Their results are combined only after both searches finish.
Useful optimizations include caching query embeddings, limiting candidate counts, batching embedding operations, and monitoring retrieval latency.
Remember that increasing candidate_k provides more candidates for RRF but also increases the amount of ranking work.
Improve Ranking Quality
RRF is a strong baseline, but it does not guarantee optimal ranking for every query.
For higher retrieval accuracy, consider adding a Cross-Encoder reranking stage after hybrid retrieval. A reranker evaluates each query-document pair and reorders the candidate passages based on relevance.
Evaluate improvements using representative queries rather than relying on a few successful examples.
Preserve Metadata and Access Controls
Production retrieval results should include document identifiers, source references, and relevant metadata. This supports citations, debugging, and document filtering.
Apply user permissions and document access restrictions before returning retrieved content to the language model. Rank fusion must never bypass these controls.
With coordinated indexing, efficient retrieval, and continuous evaluation, hybrid search can become a reliable component of a production RAG architecture.
Frequently Asked Questions
What is hybrid search in RAG?
Hybrid search in RAG combines keyword-based retrieval with semantic vector search. It helps retrieve documents that match exact search terms as well as passages that are semantically related to the user’s query.
Why combine BM25 and FAISS?
BM25 is effective at matching exact keywords, product identifiers, and technical terminology. FAISS enables semantic similarity search using vector embeddings. Combining their results can improve retrieval coverage across different query types.
Is hybrid search better than vector search?
Not always. Hybrid search can improve relevance when documents contain both specialized terminology and natural-language explanations. However, its effectiveness depends on the dataset, query patterns, embedding model, and ranking configuration.
What is Reciprocal Rank Fusion in RAG?
Reciprocal Rank Fusion (RRF) combines multiple ranked result lists using document positions rather than raw relevance scores. This makes it possible to merge BM25 and vector search results without normalizing their different scoring scales.
What is the best RRF k value?
A commonly used starting value is 60. Smaller values emphasize differences between top-ranked positions, while larger values reduce their influence. The best setting should be determined through retrieval evaluation.
Can hybrid search work without LangChain?
Yes. You can implement hybrid search for RAG directly in Python using libraries such as rank-bm25, FAISS, and Sentence Transformers. LangChain is optional and is not required for retrieval or rank fusion.
Can hybrid search be combined with reranking?
Yes. A common architecture uses BM25 and FAISS to retrieve candidate documents, RRF to combine their rankings, and a Cross-Encoder to rerank the final candidate set before passing context to the language model.
Conclusion
Building hybrid search for RAG in Python allows us to combine the strengths of keyword-based and semantic retrieval within a single search pipeline.
In this tutorial, we implemented BM25 keyword search, FAISS vector search, and Reciprocal Rank Fusion (RRF). We then combined these components into a reusable hybrid retriever and demonstrated how to evaluate retrieval quality using Recall@K.
The main advantage of hybrid retrieval is its ability to handle different query types. BM25 identifies exact terms and technical identifiers, while FAISS retrieves semantically related passages even when their wording differs from the original query.
However, combining retrieval methods does not automatically guarantee better results. Reliable RAG systems require consistent indexing, representative evaluation datasets, and continuous ranking optimization.
The next step is reranking: using a Cross-Encoder to evaluate retrieved passages more precisely before sending them to a language model. Together, hybrid retrieval and reranking provide a strong foundation for accurate, scalable RAG systems.