Retrieval-Augmented Generation (RAG) is the pattern behind most serious production AI features: the model answers questions using your documents instead of whatever it memorized during training. But the standard chunk-and-embed pipeline has a well-known weakness — it retrieves passages that look similar to the question, not necessarily passages that answer it.
Hybrid search fixes that by combining two retrieval methods — keyword (BM25) and semantic (vector) search — and it's surprisingly little code. In this guide, you'll build a production-grade hybrid RAG chatbot in Python, step by step, with working code for every stage.
By the end you'll have a chatbot that answers questions about your own documents, and you'll understand exactly why each piece exists. (New to RAG entirely? Our earlier RAG chatbot tutorial covers the basic non-hybrid pipeline — this guide upgrades it.)
Why Hybrid Search Beats Vector-Only RAG
Pure vector search embeds text into a space where semantic similarity becomes geometric proximity. That's magic for paraphrases ("how do I get my money back?" ≈ "refund policy") but weak at exact terms: product codes, error messages, function names, acronyms. Embeddings blur ERR_CONN_RESET into generic "connection trouble" territory.
Keyword search (BM25, the algorithm behind classic search engines) is the opposite: razor-sharp on exact terms, blind to synonyms and paraphrase.
Real user questions mix both needs. Hybrid search runs both retrievers in parallel and fuses the rankings, so exact identifiers pin down the right chunk while semantic matching catches rephrasings. The standard fusion method is Reciprocal Rank Fusion (RRF) — you'll implement it in ~10 lines below.
What You'll Need
- Python 3.10+
- A Qubax API key (grab one at qubax.ai — one key gives you chat models and embeddings through an OpenAI-compatible endpoint)
- The usual suspects:
requests,numpy
We'll avoid heavy framework dependencies so every step is visible.
pip install requests numpy rank-bm25Step 1: Prepare and Chunk Your Documents
Good chunking is 80% of RAG quality. The rule: chunks should be small enough to be specific but big enough to be self-contained. For prose, 400–800 tokens with ~15% overlap works well; keep paragraph boundaries intact.
def chunk_text(text: str, chunk_size: int = 1600, overlap: int = 200) -> list[str]:
"""Chunk by characters (~4 chars ≈ 1 token), respecting paragraph breaks."""
paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
chunks, current = [], ""
for para in paragraphs:
if len(current) + len(para) > chunk_size and current:
chunks.append(current.strip())
current = current[-overlap:] # keep overlap for continuity
current += para + "\n\n"
if current.strip():
chunks.append(current.strip())
return chunks
docs = [chunk_text(open(f, encoding="utf-8").read()) for f in DOC_PATHS]
chunks = [c for d in docs for c in d]Step 2: Build the Semantic Index (Embeddings)
Embed every chunk once and store the vectors. With Qubax's OpenAI-compatible API, this is a plain HTTP call:
import requests, numpy as np
QUBAX_BASE = "https://api.qubax.ai/v1" # OpenAI-compatible
API_KEY = "your-qubax-key"
def embed(texts: list[str]) -> np.ndarray:
r = requests.post(
f"{QUBAX_BASE}/embeddings",
headers={"Authorization": f"Bearer {API_KEY}"},
json={"model": "text-embedding-3-small", "input": texts},
)
r.raise_for_status()
return np.array([d["embedding"] for d in r.json()["data"]])
chunk_vectors = embed(chunks)
# normalize so cosine similarity = dot product (fast with numpy)
chunk_vectors = chunk_vectors / np.linalg.norm(chunk_vectors, axis=1, keepdims=True)Step 3: Build the Keyword Index (BM25)
One line with rank-bm25 — plus minimal tokenization:
from rank_bm25 import BM25Okapi
tokenized = [c.lower().split() for c in chunks]
bm25 = BM25Okapi(tokenized)In production you'd swap the whitespace tokenizer for something smarter (or use a real search engine like OpenSearch), but BM25 + lowercase is fine to learn on.
Step 4: The Hybrid Retrieval — Reciprocal Rank Fusion
RRF ignores raw scores (which aren't comparable between BM25 and cosine similarity) and uses only rank positions. Each retriever votes for its top results; a chunk's fused score is the sum of 1 / (k + rank) across retrievers, with k=60 as the standard constant:
def hybrid_retrieve(query: str, top_k: int = 5) -> list[int]:
# --- semantic leg ---
qv = embed([query])[0]
qv = qv / np.linalg.norm(qv)
sem_scores = chunk_vectors @ qv
sem_ranking = np.argsort(-sem_scores)[:50]
# --- keyword leg ---
kw_scores = bm25.get_scores(query.lower().split())
kw_ranking = np.argsort(-kw_scores)[:50]
# --- Reciprocal Rank Fusion ---
K = 60
fused = {}
for rank, idx in enumerate(sem_ranking):
fused[idx] = fused.get(idx, 0) + 1 / (K + rank + 1)
for rank, idx in enumerate(kw_ranking):
fused[idx] = fused.get(idx, 0) + 1 / (K + rank + 1)
ranked = sorted(fused, key=fused.get, reverse=True)
return ranked[:top_k]Chunks retrieved by both legs bubble to the top — exactly the behavior you want.
Step 5: Wire Up the Chatbot
Pull the top chunks, stuff them into the prompt, and call a chat model:
def ask(question: str) -> str:
top = hybrid_retrieve(question, top_k=5)
context = "\n\n---\n\n".join(chunks[i] for i in top)
r = requests.post(
f"{QUBAX_BASE}/chat/completions",
headers={"Authorization": f"Bearer {API_KEY}"},
json={
"model": "gpt-5.4-mini",
"messages": [
{"role": "system", "content":
"Answer using ONLY the provided context. "
"If the answer isn't in the context, say you don't know."},
{"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
],
},
)
r.raise_for_status()
return r.json()["choices"][0]["message"]["content"]
print(ask("What is our refund policy for annual plans?"))Step 6: Test It Like an Engineer
Build a tiny eval set — 15–20 real questions with known source chunks — and measure retrieval hit rate (is the right chunk in the top 5?). Compare three configurations: keyword-only, vector-only, hybrid. On documentation-style corpora you should typically see hybrid win by 10–25 points of hit rate, with the biggest gains on questions containing exact identifiers.
Production Hardening Checklist
- Persist your indexes — embeddings in a vector store, BM25 in an inverted index; rebuild only on document change
- Add a reranker — retrieve 50 candidates hybrid-style, then rerank the top 20 with a cross-encoder for another quality bump
- Log retrieval failures — every "I don't know" answer is either a coverage gap or a retrieval bug; both are gold
- Cache embeddings by content hash so re-chunking doesn't re-bill the whole corpus
- Watch cost per query — routing easy queries to a mini-class model can cut chat costs ~90%; see the Qubax model catalog for pricing side-by-sides
Wrapping Up
You now have a complete hybrid RAG chatbot: chunking, dual retrieval (BM25 + embeddings), RRF fusion, and grounded generation — all in under 150 lines of dependency-light Python. The hybrid pattern is the single highest-leverage upgrade you can make to a vector-only RAG stack, and it composes cleanly with everything else (rerankers, metadata filters, query rewriting).
Ready to build? Get your API key at [qubax.ai](https://qubax.ai), explore models at [qubax.ai/models](https://qubax.ai/models), and check the full API reference at [qubax.ai/docs](https://qubax.ai/docs).
FAQ
What is hybrid search in RAG?
Hybrid search combines keyword retrieval (BM25) with semantic vector retrieval and merges the results — usually with Reciprocal Rank Fusion. Exact terms match via keywords, paraphrases match via embeddings, and the fused ranking is better than either alone.
Why not just use vector search?
Vector embeddings are fuzzy about exact tokens — error codes, product SKUs, function names, and rare acronyms get semantically "smoothed" away. Keyword search nails those, which is why fusing the two consistently improves retrieval quality.
What is Reciprocal Rank Fusion (RRF)?
A simple formula that merges ranked lists using only rank positions: each result scores 1/(k + rank) from each list, summed across lists. It needs no score calibration between retrievers, and k=60 works well in practice.
How big should my chunks be?
Typically 400–800 tokens with 10–20% overlap, cutting at paragraph boundaries. Too small loses context; too big dilutes relevance and wastes context-window budget. Empirically test two or three sizes on your own documents.
Can I use this pattern with any LLM provider?
Yes — the code uses Qubax's OpenAI-compatible endpoints, and the same pattern works against any provider. The advantage of Qubax is swapping chat or embedding models is a one-line change, with no per-provider SDK rewrites.
