Step-by-Step Guide: How I Built and Scaled a Private RAG Pipeline with Ollama and QdrantA few months ago, our engineering team faced a strict regulatory barrier. We were tasked with building an internal intelligence portal designed to query hundreds of thousands of highly sensitive corporate documents, intellectual property papers, and financial audits. Sending this data to cloud LLM APIs was a complete non-starter due to compliance rules. We had to build something local, fast, and highly reliable.In this guide, I will share the exact blueprint we used to design, deploy, and scale a private Retrieval-Augmented Generation (RAG) pipeline in production. By combining Ollama for lightweight local inference, Qdrant as our lightning-fast vector database, and LlamaIndex as the orchestration engine, we achieved sub-second latency and absolute data privacy.The Core ArchitectureA standard RAG pipeline works by converting unstructured text into mathematical representations called vectors, storing those vectors in a specialized index, and querying them based on semantic similarity. When a user asks a question, the system retrieves the most relevant document chunks and passes them along with the original question to a local Large Language Model (LLM) to generate a grounded, hallucination-free response.For our production setup, we selected the following software stack:Inference Engine: Ollama running a quantized version of Llama-3.1-8B.Embedding Model: nomic-embed-text due to its exceptional 8192 token context window.Vector Store: Qdrant, running as a containerized cluster.Framework: LlamaIndex (Python) to coordinate document parsing, embedding generation, and vector index matching.Step 1: Setting Up the Local InfrastructureTo run everything locally, we utilize Docker Compose. This ensures environment parity between developer workstations and our target private bare-metal servers. Below is the configuration file we used to launch both Qdrant and Ollama with GPU acceleration enabled.services:
qdrant:
image: qdrant/qdrant:latest
container_name: qdrant-db
ports:
- "6333:6333"
- "6334:6334"
volumes:
- ./qdrant_storage:/qdrant/storage
restart: always
ollama:
image: ollama/ollama:latest
container_name: ollama-service
ports:
- "11434:11434"
volumes:
- ./ollama_storage:/root/.ollama
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
restart: alwaysAfter saving this as docker-compose.yml, spin up the stack with the command: docker compose up -d. This launches Qdrant with persistent storage mapped locally and binds Ollama to utilize available NVIDIA GPUs.Step 2: Pulling and Configuring Local ModelsNow that Ollama is up and running, we need to download our foundation LLM and our specialized Embedding Model. Executing the commands inside the running Ollama container downloads the weights directly to your local persistent storage:docker exec -it ollama-service ollama pull llama3.1:8b
docker exec -it ollama-service ollama pull nomic-embed-textWe chose nomic-embed-text because it excels at local document chunk classification, outputting a high-quality 768-dimensional space which aligns perfectly with modern search requirements. Llama3.1-8B serves as our generation engine, strikes an excellent balance between low latency and complex logical reasoning, and operates fully offline.Step 3: Building the Python Ingestion PipelineWith infrastructure ready, we write our core data ingestion service. The objective is to monitor a directory of sensitive documents, automatically chunk them into digestible pieces, compute vector embeddings, and index them inside Qdrant. First, ensure your environment has the required packages installed:pip install llama-index llama-index-vector-stores-qdrant qdrant-client llama-index-llms-ollama llama-index-embeddings-ollamaNext, we construct the ingestion code. In our experience, setting an aggressive chunk size with a generous overlap yields the best retrieval accuracy when dealing with complex corporate policies.import os
from qdrant_client import QdrantClient
from llama_index.core import SimpleDirectoryReader, StorageContext, VectorStoreIndex, Settings
from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding
Configure our local LLM and embedding configurations
Settings.llm = Ollama(model="llama3.1:8b", base_url="http://localhost:11434", request_timeout=120.0) Settings.embed_model = OllamaEmbedding( model_name="nomic-embed-text", base_url="http://localhost:11434", embed_batch_size=32 )def ingest_documents(dir_path: str):
# 1. Initialize our Qdrant client
client = QdrantClient(url="http://localhost:6333")
# 2. Create the vector database schema integration
vector_store = QdrantVectorStore(client=client, collection_name="internal_knowledge")
storage_context = StorageContext.from_defaults(vector_store=vector_store)
# 3. Read our local documents folder
print(f"Reading documents from {dir_path}...")
documents = SimpleDirectoryReader(dir_path).load_data()
# 4. Generate embeddings and populate the database
print("Generating local embeddings and building index...")
index = VectorStoreIndex.from_documents(
documents,
storage_context=storage_context,
show_progress=True
)
print("Ingestion complete! All data successfully vectorized.")
return index
if __name__ == "__main__":
# Create target directory for demonstration
os.makedirs("./private_docs", exist_ok=True)
ingest_documents("./private_docs")Step 4: Building the Conversational Query EngineNow that the knowledge base is fully populated, we need to query it. Instead of general question-answering, we enforce strict grounding rules via the LLM system prompt. This ensures that the engine only answers questions based on retrieved knowledge and explicitly declares "I don't know" rather than hallucinating facts.The following query handler establishes a secure connection to Qdrant, dynamically retrieves relevant source vectors, assembles context, and streams the output directly to the console for a fluid UX.def run_query_engine(user_query: str):
client = QdrantClient(url="http://localhost:6333")
vector_store = QdrantVectorStore(client=client, collection_name="internal_knowledge")
index = VectorStoreIndex.from_vector_store(vector_store=vector_store)
# Define the strict behavior we want via the system prompt
system_prompt = (
"You are a secure, closed-domain enterprise AI assistant. "
"You must answer the user query based ONLY on the provided context below. "
"If the answer is not mentioned in the context, clearly state that "
"you do not possess that information inside the provided database. "
"Do not make up facts or use external knowledge. Keep answers precise."
)
query_engine = index.as_query_engine(
similarity_top_k=4,
system_prompt=system_prompt,
streaming=True
)
response = query_engine.query(user_query)
print("\n--- RESPONSE ---")
response.print_response_stream()
print("\n----------------\n")
if __name__ == "__main__":
# Run query test
run_query_engine("What are the specific quarterly targets outlined in the local roadmap?")Step 5: Optimizing for ProductionWhile the initial implementation worked perfectly on high-end desktop hardware, scaling this pipeline for multi-user production workloads on enterprise servers revealed several bottleneck vectors. Here are the optimizations we applied to maintain efficiency:1. Vector Search Performance TuningBy default, Qdrant relies on standard index sweeps. As your database scale surpasses 100,000 document segments, construct a custom HNSW index configuration. Tuning HNSW settings allows you to trade a fraction of a percent of accuracy for massive gains in search speed. Modify the collection parameters with Qdrant to optimize construction parameters.2. Semantic Chunk OverlapUsing a default parser blindly breaks down paragraphs without respecting logic limits. We solved this by using LlamaIndex's SentenceSplitter with a chunk size of 512 tokens and a 10% overlap (50 tokens). This guarantees that context remains intact across boundaries, directly reducing fragmented, meaningless search hits.3. Leveraging Quantized ModelsRunning full 16-bit float models requires massive VRAM overhead. Through careful testing, we converted our generation steps to a 4-bit Quantization schema (using Ollama's default Q4_K_M). This dropped our memory footprint by nearly 70% with negligible loss in reasoning ability, enabling us to run inference easily on standard enterprise Nvidia RTX workstation GPUs.Frequently Asked Questions (FAQs)Frequently Asked Questions (FAQs)How does this setup ensure complete data privacy?Since both Ollama and Qdrant are running inside your own container network (Docker), no network packets containing document contents, user queries, or vector calculations leave your physical server environment. You can physically unplug your internet connection and the entire system will continue to work perfectly.What kind of hardware specs are required to scale this locally?For a team of up to 50 concurrent users, we found that a single dedicated workstation running a 24GB VRAM NVIDIA RTX 3090/4090 or RTX A5000, paired with 64GB of system RAM and an NVMe SSD, was more than sufficient to host both the model and the vector search layers with sub-second response times.Can I easily add hybrid search to this stack?Yes, Qdrant natively supports hybrid search combining semantic vector search with keyword matching (BM25). You can configure LlamaIndex to use Qdrant's sparse vectors alongside dense embeddings, providing a dual-search interface that handles technical codes and names much better than semantic match alone.How do I handle document updates and deletions?LlamaIndex and Qdrant track unique hashes of document nodes. When you update your internal folder, you can run a differential ingestion script that reads files, identifies which file hashes have changed, deletes the outdated points from Qdrant by filtering against the source file ID metadata, and indexes only the newly modified segments.
