Step-by-Step Guide: How We Built a Production-Grade Hybrid RAG Pipeline with Qdrant and Cohere

Step-by-Step Guide: How We Built a Production-Grade Hybrid RAG Pipeline with Qdrant and Cohere

In my experience, moving Retrieval-Augmented Generation (RAG) from a naive local prototype to a production system that handles millions of documents is where most engineering teams hit a wall. When we first configured a basic vector search pipeline using cosine similarity on OpenAI embeddings, we noticed a major issue: the retrieval engine was great at capturing semantic concepts, but terrible at finding precise keywords, serial numbers, or specific product codes. Our users were frustrated by the lack of exact-match precision.

To solve this, we redesigned our architecture around Hybrid Search (combining dense and sparse vectors) and introduced a secondary Reranking stage using Cohere. In this guide, I will walk you through the exact implementation steps, architectural choices, and configuration snippets we used to build a robust, scalable, and highly accurate RAG system.

Step 1: Architecting the Hybrid RAG Pipeline

Before writing code, we must understand why standard vector search falls short. Pure vector search relies on dense embeddings (like those from OpenAI or Hugging Face). These models compress semantic meaning into a continuous vector space. However, they struggle with exact keyword matching, technical specifications, and domain-specific acronyms.

To overcome this, we adopted a hybrid retrieval model containing three major phases:

Dense Retrieval: Leveraging dense embeddings for semantic, contextual similarity.
Sparse Retrieval: Utilizing token-based sparse vectors (similar to BM25 but modernized via learned sparse representations) to capture exact keyword occurrences.
Cross-Encoder Reranking: Passing the top candidates through a deep learning reranker to calculate a highly accurate relevance score between the query and each document.

By delegating the initial candidate generation to the vector database and using a highly accurate transformer to sort only the top 20 to 50 results, we optimized both search precision and query latency.

Step 2: Setting Up the Infrastructure with Qdrant

In our production testing, we selected Qdrant as our vector search engine. It natively supports hybrid search, allows us to store both dense and sparse vectors in a single point, and features excellent payload filtering. Below is the Python code using the official client to initialize a collection with both dense and sparse configurations.

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, SparseVectorParams, SparseIndexParams

Initialize local or cloud client

client = QdrantClient(host="localhost", port=6333)

COLLECTION_NAME = "kb_hybrid_articles"

Define configuration for both vector types

client.recreate_collection( collection_name=COLLECTION_NAME, vectors_config={ "dense-embeddings": VectorParams( size=1536, # Standard size for OpenAI text-embedding-3-small distance=Distance.COSINE ) }, sparse_vectors_config={ "sparse-keywords": SparseVectorParams( index=SparseIndexParams( on_disk=True ) ) } ) print(f"Collection '{COLLECTION_NAME}' successfully initialized!")

Here, the HNSW index manages our dense search structures on disk, while a separate inverted index handles our sparse, keyword-focused weights.

Step 3: Implementing Text Chunking and Vector Ingestion

When indexing documents, the size of your raw text blocks matters immensely. In our production pipeline, we settled on a recursive character chunker with a size of 512 characters and an overlap of 64 characters. This preserves enough contextual vocabulary without overloading the vector space.

We use OpenAI for dense embeddings and a lightweight sparse encoder (like BM25 or FastEmbed's SPLADE) to generate sparse tokens. Below is how we prepare and upload our data payload to Qdrant:

import openai
from qdrant_client.models import PointStruct

Configure your API keys safely

openai.api_key = "your_openai_api_key_here"

def get_dense_embedding(text: str) -> list[float]:
response = openai.Embedding.create(
input=[text],
model="text-embedding-3-small"
)
return response['data'][0]['embedding']

def get_sparse_embedding(text: str) -> dict:
# Simple tokenization for sparse generation (BM25 mock-up for explanation)
words = text.lower().split()
indices = [hash(word) % 100000 for word in words]
values = [words.count(word) / len(words) for word in words]
return {"indices": indices, "values": values}

Ingesting dummy data containing both exact serial numbers and semantic content

documents = [ { "id": 1, "text": "Our API gateway server has a serial number of SN-9983-X. Maintain configuration via settings.json.", "metadata": {"category": "devops"} }, { "id": 2, "text": "To fix connection timeouts, adjust the maximum pooling size on your PostgreSQL client pool settings.", "metadata": {"category": "database"} } ]

points = []
for doc in documents:
dense = get_dense_embedding(doc["text"])
sparse = get_sparse_embedding(doc["text"])

points.append(PointStruct(
id=doc["id"],
vector={
"dense-embeddings": dense,
"sparse-keywords": sparse
},
payload={
"text": doc["text"],
"metadata": doc["metadata"]
}
))

client.upsert(
collection_name=COLLECTION_NAME,
points=points
)
print("Data successfully ingested with dual vector index.")

This hybrid ingestion process ensures that both the exact phrase properties (like SN-9983-X) and the contextual concepts (like database pooling) are fully discoverable.

Step 4: Executing the Hybrid Query and Cohere Rerank

Once your indexes are loaded with documents, the retrieval step queries both the dense and sparse models concurrently. In our initial design, we tried combining dense and sparse distances using Reciprocal Rank Fusion (RRF) directly inside the database query. However, RRF is highly sensitive to the weights assigned to each search method.

To eliminate manual tuning, we introduced a Cohere Rerank model. In this setup, we retrieve a broader set of candidate documents (e.g., 50 candidates) using both vector styles and feed them along with the search query to Cohere's endpoint. The cross-encoder model scores candidate relevance far more accurately than standard dot-product mathematics ever could.

Here is our implementation script for the complete end-to-end retrieval phase:

import cohere
from qdrant_client.models import NamedVector, NamedSparseVector

co = cohere.Client("your_cohere_api_key_here")

def hybrid_search_and_rerank(query: str, limit: int = 10) -> list[dict]:
# 1. Fetch dense vector representation
query_dense = get_dense_embedding(query)

# 2. Fetch sparse vector representation
query_sparse = get_sparse_embedding(query)

# 3. Query Qdrant with both vector models
results = client.search_batch(
collection_name=COLLECTION_NAME,
requests=[
client.models.SearchRequest(
vector=NamedVector(name="dense-embeddings", vector=query_dense),
limit=limit * 2
),
client.models.SearchRequest(
vector=NamedSparseVector(name="sparse-keywords", vector=query_sparse),
limit=limit * 2
)
]
)

# Flatten unique candidate items retrieved
seen_ids = set()
candidates = []
for batch in results:
for hit in batch:
if hit.id not in seen_ids:
seen_ids.add(hit.id)
candidates.append(hit.payload["text"])

if not candidates:
return []

# 4. Perform secondary reranking with Cohere
rerank_response = co.rerank(
query=query,
documents=candidates,
top_n=limit,
model="rerank-english-v3.0"
)

ranked_results = []
for item in rerank_response.results:
ranked_results.append({
"document": candidates[item.index],
"score": item.relevance_score
})

return ranked_results

Test retrieval for an exact serial number

results = hybrid_search_and_rerank("Which server has serial SN-9983-X?", limit=2) for idx, res in enumerate(results): print(f"Rank {idx+1}: {res['document']} (Score: {res['score']:.4f})")

Step 5: Optimizing Performance, Latency, and Cost

When running this architecture at scale in production, latency overhead is your biggest challenge. The dense embedding API, Qdrant cluster queries, and Cohere's reranking endpoint each introduce round-trip times. In our initial test phases, typical requests hovered around 450ms. Here is how we managed to slash that down to less than 120ms:

Implement Embedding Caching: For common queries, we cached embeddings in Redis with an expiration duration of 24 hours. If a user asked a query that had been analyzed recently, we bypassed OpenAI completely.
Batching Vector Operations: Avoid sending single payload writes to your vector DB. Collect records and write them in chunks of 100 to 500 records to minimize network connection handshakes.
Filtering Prefetches: Always use metadata filters directly within the Qdrant query step rather than filtering the results afterward in your application code. This reduces index search times.
Locality of Reranker: If you are working with strict latency bounds, consider hosting an open-source cross-encoder locally using vLLM or Hugging Face's TEI (Text Embeddings Inference) server. This avoids the cloud-routing lag of third-party model providers.

By executing these steps, we achieved a production pipeline that delivers both contextual accuracy and exact keyword relevance, allowing our RAG layer to feed the downstream LLM with only the highest-quality chunks.

Frequently Asked Questions (FAQs)

• Why do we need hybrid search instead of pure dense vector search?
• Pure dense vector search relies on mathematical semantic proximity. It functions wonderfully for queries seeking broad answers, but fails when a user searches for precise serial numbers, system error codes, or domain-specific identifiers. Hybrid search integrates a keyword-focused sparse vector layer to capture exact-match tokens alongside semantic meanings.

• How does a cross-encoder reranker differ from standard embedding models?
• Standard embedding models encode the document and the query separately, calculating a basic similarity score using simple geometry. A cross-encoder, such as Cohere Rerank, evaluates both the query and the document simultaneously using attention layers. This allows the transformer to evaluate interactive relationships between tokens, resulting in significantly higher accuracy at the cost of additional compute. That is why it is used as a secondary step on a small subset of candidate documents.

• How do you handle cold starts or frequent updates to document indices?
• Since Qdrant operates as an online vector database, new vectors can be continuously upserted without locking search operations. In production, we run background cron jobs to process modified internal articles, generate both vector models, and commit upserts to the collection asynchronously to avoid impacting incoming user traffic.

• Is it possible to host the reranking step on our own secure private servers?
• Yes. If security or latency requirements prevent you from utilizing Cohere's hosted cloud services, you can deploy a model from the BAAI family (such as BGE-Reranker) locally inside a Docker container using Hugging Face's TEI tool. This provides high-throughput cross-encoder evaluation with low operational overhead.