Build a Documentation-Grounded AI Coding Assistant in Python (That Doesn’t Hallucinate APIs)

Most AI coding assistants die in one of two ways: they hallucinate an API that doesn't exist, or they confidently edit the wrong file. The fix isn't a better model — it's grounding your agent in real, live documentation and codebase context. In this tutorial you'll build a documentation-grounded coding assistant in Python that answers from your actual stack, not from its training data's stale memories.

What You'll Build

By the end of this guide you'll have a working CLI coding assistant that:

1. Ingests your project's documentation (markdown files, API docs) into a local vector index
2. Retrieves the relevant chunks for each user question before the model answers
3. Cites its sources — every answer links back to the doc page it used
4. Falls back gracefully when retrieval finds nothing relevant, instead of hallucinating

Total cost to run: fractions of a cent per question on a cheap model. Total time: about 45 minutes.

Prerequisites

• Python 3.11+
• An API key for a chat model and a cheap embedding model (grab one on Qubax — any OpenAI-compatible endpoint works)
• A folder of markdown docs to index (your company wiki export, library docs, etc.)

pip install openai numpy

Configure credentials the standard way: set the OPENAI_API_KEY environment variable to your key, and OPENAI_BASE_URL to your provider's endpoint (for example https://api.qubax.ai/v1). See the Qubax docs for details.

Step 1: Chunk the Documentation

The #1 beginner mistake is embedding whole pages. Retrieval quality collapses because one page covers five topics. The fix: split on markdown headers so each chunk is one coherent idea.

# chunker.py
import re
from pathlib import Path

def chunk_markdown(text: str, source: str, max_chars: int = 1500):
    """Split markdown into chunks at headers, keeping the header trail as context."""
    lines = text.split("\n")
    chunks, current, headers = [], [], []
    for line in lines:
        m = re.match(r"^(#{1,4})\s+(.*)", line)
        if m:
            if current:
                chunks.append((" > ".join(headers), "\n".join(current).strip()))
                current = []
            depth = len(m.group(1))
            headers = headers[:depth - 1] + [m.group(2).strip()]
        current.append(line)
    if current:
        chunks.append((" > ".join(headers), "\n".join(current).strip()))

    out = []
    for headers_trail, body in chunks:
        if not body:
            continue
        # split oversized chunks on paragraphs
        while len(body) > max_chars:
            cut = body.rfind("\n\n", 0, max_chars) or max_chars
            out.append({"source": source, "section": headers_trail, "text": body[:cut]})
            body = body[cut:]
        if body.strip():
            out.append({"source": source, "section": headers_trail, "text": body})
    return out

def load_docs(docs_dir: str):
    chunks = []
    for path in Path(docs_dir).rglob("*.md"):
        text = path.read_text(encoding="utf-8", errors="ignore")
        chunks.extend(chunk_markdown(text, source=path.name))
    return chunks

Each chunk keeps its header trail (e.g. "Authentication > API Keys > Rotating Keys"), which dramatically improves embedding quality — the chunk's embedding captures what section of the doc it lives in.

Step 2: Build the Vector Index

We'll use a cheap embedding model and store vectors in plain NumPy. For a few thousand chunks, a brute-force cosine search is faster than you'd think (milliseconds) and requires zero infrastructure.

# indexer.py
import json, numpy as np
from openai import OpenAI
from chunker import load_docs

client = OpenAI()  # reads credentials from the environment
EMBED_MODEL = "text-embedding-3-small"  # any cheap embedding model

def embed(texts: list[str]) -> np.ndarray:
    resp = client.embeddings.create(model=EMBED_MODEL, input=texts)
    return np.array([d.embedding for d in resp.data], dtype=np.float32)

def build_index(docs_dir: str, out_path: str = "index.npz"):
    chunks = load_docs(docs_dir)
    print(f"Indexing {len(chunks)} chunks...")
    vectors = embed([c["text"] for c in chunks])
    vectors /= np.linalg.norm(vectors, axis=1, keepdims=True)  # normalize for cosine
    np.savez_compressed(out_path, vectors=vectors,
                        meta=json.dumps(chunks).encode())
    print(f"Saved {out_path}")

if __name__ == "__main__":
    build_index("docs/")

Run it once (and again whenever docs change — or wire it into CI):

python indexer.py

Step 3: Retrieve With a Similarity Floor

Here's where we prevent hallucination #1: if the best match is weak, say so. Don't feed the model marginal context and hope.

# retriever.py
import json, numpy as np
from indexer import embed

def retrieve(query: str, index_path="index.npz", top_k=4, min_score=0.30):
    data = np.load(index_path)
    vectors = data["vectors"]
    chunks = json.loads(bytes(data["meta"]).decode())

    q = embed([query])[0]
    q /= np.linalg.norm(q)
    scores = vectors @ q
    idx = np.argsort(-scores)[:top_k]

    results = [(float(scores[i]), chunks[i]) for i in idx if scores[i] >= min_score]
    if not results:
        return [], None  # signal: nothing relevant found
    return results, scores[idx[0]]

Tune min_score per embedding model — 0.25–0.35 is a reasonable starting range for cosine similarity with normalized embeddings. Log your scores on real queries and adjust.

Step 4: The Grounded Assistant Loop

Now the chat model. The prompt has three hard rules that keep answers anchored, plus an explicit escape hatch:

# assistant.py
from openai import OpenAI
from retriever import retrieve

client = OpenAI()
CHAT_MODEL = "gpt-5.6-luna"   # a cheap, fast model — see https://qubax.ai/models

SYSTEM = """You are a coding assistant for our internal stack.
Rules:
1. Answer ONLY from the provided documentation context.
2. Cite the source file and section for every factual claim, like [auth.md > API Keys].
3. If the context doesn't cover the question, say exactly:
   "I couldn't find this in the documentation." Then suggest where to look.
Never invent API names, parameters, or configuration options."""

def ask(question: str) -> str:
    results, top_score = retrieve(question)
    if not results:
        return "I couldn't find this in the documentation. Try checking the docs folder or asking a human."

    context = "\n\n".join(
        f"### Source: {c['source']} > {c['section']}\n{c['text']}"
        for _, c in results
    )
    resp = client.chat.completions.create(
        model=CHAT_MODEL,
        temperature=0.1,
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": f"Documentation context:\n\n{context}\n\nQuestion: {question}"},
        ],
    )
    return resp.choices[0].message.content

if __name__ == "__main__":
    import sys
    print(ask(" ".join(sys.argv[1:])))

python assistant.py "How do I rotate API keys without downtime?"

Step 5: Level It Up Into a Coding Agent

A doc answerer is useful; a coding agent is a force multiplier. Two extensions, in increasing difficulty:

5a. Add your actual code to the index

Index your own source files with the same pipeline (split on function/class boundaries instead of headers). Now questions like "where do we validate webhook signatures?" return real code with file paths — and the model's suggested edits reference functions that actually exist.

5b. Give the model tools

Add a function-calling loop with two tools: search_docs(query) and read_file(path). The model decides when to look things up. The critical safety property: the tools can only read. Never give a grounding agent write access in its first iteration — write access plus hallucinated assumptions is how agent incidents happen.

tools = [
    {"type": "function", "function": {
        "name": "search_docs",
        "description": "Search internal documentation. Use before answering any question about APIs or conventions.",
        "parameters": {"type": "object", "properties": {
            "query": {"type": "string"}}, "required": ["query"]}}},
    {"type": "function", "function": {
        "name": "read_file",
        "description": "Read a source file from the repository.",
        "parameters": {"type": "object", "properties": {
            "path": {"type": "string"}}, "required": ["path"]}}},
]

Loop on finish_reason == "tool_calls", execute each tool, append results as role: "tool" messages, and call the model again. Keep a max of ~8 tool rounds to bound cost.

Cost Check: What This Actually Costs

With a cheap chat model and a cheap embedding model, a typical question costs:

• 1 embedding call (~100 tokens): effectively free
• 1 chat call with ~2,000 tokens of context + ~300 output tokens: well under half a cent

You can drop this further with prompt caching — since your system prompt is static and doc chunks repeat across questions, caching cuts input costs dramatically on conversational follow-ups. Check live per-token prices for cheap models like GPT-5.6 Luna, Gemini Flash variants, or GLM Flash on qubax.ai/models and pick whichever is cheapest for your volume.

Common Pitfalls

• Chunk overlap done wrong. Small overlaps (100-200 chars) help at boundaries; huge overlaps bloat context and duplicate citations.
• Stale index. Docs change. Rebuild on merge, or your assistant confidently cites deleted endpoints — grounded hallucination is still hallucination.
• Citations that lie. If the model cites a section that wasn't in the retrieved context, tighten the system prompt and lower temperature to 0.
• One model for everything. Retrieval + cheap model handles 90% of questions. Route only the hard architectural questions to a frontier reasoning model.

FAQ

Why not just paste the docs into the prompt?

Context windows are big but not free, and retrieval quality drops with a flooded context. RAG gives you a tiny, relevant, cited context — cheaper per query and more accurate. Use a long-context paste only for docs under ~10K tokens.

Which models work best for a grounded coding assistant?

Cheap and fast beats fancy here: GPT-5.6 Luna-class, Gemini Flash, GLM Flash, or DeepSeek Flash variants all handle grounded Q&A well. Save reasoning models for architecture questions. Compare live pricing on qubax.ai/models.

How often should I rebuild the index?

On every docs change — it takes seconds for a few thousand chunks. In CI, run python indexer.py as a post-merge step.

Can I use this with any API provider?

Yes — the code uses the OpenAI SDK format, which is compatible with Qubax and most providers. Just point OPENAI_BASE_URL at your provider and pick model names from its catalog. See the Qubax docs for endpoint details.

Build it, then make it cheap: run your grounded assistant on wholesale-priced models at [qubax.ai/models](https://qubax.ai/models) — same models, open market where compute providers compete on price.