Build a Multimodal Document Q&A API in Python: Vision + Text in 20 Minutes

Introduction

Most Q&A tutorials stop at plain text. Real documents aren't plain text: they're scanned invoices, photographed receipts, slide decks, contracts with stamped signatures, and screenshots of dashboards. If your pipeline can't see the page, it can't answer half the questions users ask.

In this tutorial you'll build a multimodal document Q&A API that:

1. Accepts an uploaded PDF or image plus a natural-language question.
2. Converts pages to images and reads them with a vision-language model (no fragile OCR preprocessing required).
3. Answers with citations back to page numbers.
4. Caches answers so identical questions don't re-bill you.
5. Tracks exact cost per request so the service never quietly burns budget.

We'll use Python, FastAPI, and an OpenAI-compatible endpoint, so the same code works with GPT-5.6, Claude, GLM, MiMo, or any model you route to. Total time: about 20 minutes.

The full code is at the bottom. Follow along section by section if you want to understand each piece.

Prerequisites

• Python 3.10+
• An API key for an OpenAI-compatible provider (we'll use Qubax — one key, any model)
• pip install fastapi uvicorn openai pypdf pillow python-multipart

For the vision step, pick a model that accepts image input. Good options at very different price points include gpt-5.6-terra (premium), glm-5v-turbo (mid-range vision specialist), or mimo-v2.5 (budget). The code below takes the model name as an environment variable so you can swap freely.

Architecture in One Paragraph

The client uploads a file and a question. We render each PDF page (or the raw image) to a JPEG, base64-encode it, and send the pages plus the question to a vision-language model in a single request. The model must answer in JSON with an answer field and a sources array of page numbers. We hash (file_bytes, question, model) as a cache key; on a hit we return the stored answer and cost $0. On a miss we call the model, log the token usage and computed cost, and store everything.

That's it — four moving parts: render, prompt, cache, meter.

Step 1: Render Documents to Images

Vision models don't read PDFs directly; they read images. Convert first:

# renderer.py
import io
from pypdf import PdfReader
from PIL import Image

def file_to_page_images(data: bytes, filename: str, max_pages: int = 8,
                        max_dim: int = 1600) -> list[str]:
    """Return a list of base64-encoded JPEGs, one per page/image."""
    import base64
    pages: list[bytes] = []

    if filename.lower().endswith(".pdf"):
        reader = PdfReader(io.BytesIO(data))
        # pip install pdf2image (requires poppler) OR rasterize with pymupdf
        import fitz  # pymupdf
        doc = fitz.open(stream=data, filetype="pdf")
        for i, page in enumerate(doc):
            if i >= max_pages:
                break
            pix = page.get_pixmap(dpi=150)
            pages.append(pix.tobytes("jpeg"))
    else:
        pages.append(data)

    out = []
    for raw in pages:
        img = Image.open(io.BytesIO(raw)).convert("RGB")
        img.thumbnail((max_dim, max_dim))  # keep payloads small
        buf = io.BytesIO()
        img.save(buf, "JPEG", quality=82)
        out.append(base64.b64encode(buf.getvalue()).decode())
    return out

Two rules that save real money: downscale before sending (a 4000px phone photo costs more tokens than a 1600px thumbnail with no accuracy gain for text reading), and cap page count with a clear error message instead of silently truncating.

Step 2: The Vision Prompt (Forcing Citations)

The single biggest quality lever in this whole system is the prompt. We want strict JSON and page-level grounding:

SYSTEM_PROMPT = """You are a precise document analyst.
You receive images of document pages, numbered in order starting at page 1.
Answer the user's question using ONLY the visible content.
Rules:
- Respond with JSON only: {"answer": str, "sources": [int], "confidence": "high"|"medium"|"low"}
- "sources" lists the page numbers that support the answer.
- If the document does not contain the answer, set confidence "low" and say so in "answer".
- Never invent numbers, dates, or names."""

Requesting JSON with structured output / json mode (supported by most OpenAI-compatible APIs) removes the classic "sure, here's your JSON:" preamble problem. Parse defensively anyway:

import json

def parse_answer(text: str) -> dict:
    try:
        obj = json.loads(text)
    except json.JSONDecodeError:
        start, end = text.find("{"), text.rfind("}")
        if start == -1:
            raise ValueError(f"Model returned non-JSON: {text[:200]}")
        obj = json.loads(text[start:end + 1])
    obj.setdefault("sources", [])
    obj.setdefault("confidence", "medium")
    return obj

Step 3: Call the Model with Images

The OpenAI-compatible image format is a list of content parts — text plus image_url entries carrying base64 data URLs:

# qa.py
import base64, hashlib, json
from openai import OpenAI
from renderer import file_to_page_images
from prompt import SYSTEM_PROMPT, parse_answer

client = OpenAI(base_url="https://api.qubax.ai/v1", api_key="YOUR_QUBAX_KEY")

def answer_document(file_bytes: bytes, filename: str, question: str,
                    model: str = "glm-5v-turbo") -> dict:
    images = file_to_page_images(file_bytes, filename)
    content = [{"type": "text",
                "text": f"{len(images)} page(s). Question: {question}"}]
    for i, b64 in enumerate(images, 1):
        content.append({"type": "text", "text": f"--- PAGE {i} ---"})
        content.append({"type": "image_url",
                        "image_url": {"url": f"data:image/jpeg;base64,{b64}"}})

    resp = client.chat.completions.create(
        model=model,
        temperature=0,
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": content},
        ],
    )
    msg = resp.choices[0].message
    result = parse_answer(msg.content or "{}")
    result["usage"] = {
        "input_tokens": resp.usage.prompt_tokens,
        "output_tokens": resp.usage.completion_tokens,
    }
    return result

Note temperature=0: document Q&A is extraction, not creative writing. Determinism also makes your cache far more effective.

Step 4: Cache and Meter Every Request

The cache is where the cost control happens. Key on the exact inputs; store the model's own usage report; convert tokens to dollars with your provider's published per-million prices:

# metering.py
import hashlib, json

PRICES = {  # USD per 1M tokens — update from your provider's pricing page
    "glm-5v-turbo": {"in": 0.83, "out": 2.77},
    "gpt-5.6-terra": {"in": 0.42, "out": 1.68},
    "mimo-v2.5": {"in": 0.08, "out": 0.16},
}

def cost_of(model: str, in_tok: int, out_tok: int) -> float:
    p = PRICES.get(model, {"in": 1.0, "out": 1.0})
    return (in_tok * p["in"] + out_tok * p["out"]) / 1_000_000

def cache_key(model: str, file_sha: str, question: str) -> str:
    return hashlib.sha256(
        json.dumps([model, file_sha, question.strip().lower()]).encode()
    ).hexdigest()

Keep the price table in configuration, not code, and refresh it when providers update. (This is also why marketplaces are convenient: Qubax's model catalog exposes per-token pricing for every model through one interface, so your metering table stays short.)

Step 5: The FastAPI Service

# app.py
import hashlib, json
from collections import defaultdict
from fastapi import FastAPI, UploadFile, Form, HTTPException
from qa import answer_document
from metering import cost_of, cache_key

app = FastAPI(title="DocQA")
CACHE: dict[str, dict] = {}
SPEND: dict[str, float] = defaultdict(float)
DAILY_BUDGET_USD = 20.0

@app.post("/ask")
async def ask(file: UploadFile, question: str = Form(...), model: str = Form("glm-5v-turbo")):
    data = await file.read()
    if len(data) > 15_000_000:
        raise HTTPException(413, "File too large (15MB limit)")

    key = cache_key(model, hashlib.sha256(data).hexdigest(), question)
    if key in CACHE:
        return {**CACHE[key], "cached": True, "cost_usd": 0.0}

    if SPEND["day"] >= DAILY_BUDGET_USD:
        raise HTTPException(429, "Daily budget exhausted")

    result = answer_document(data, file.filename or "doc", question, model)
    cost = cost_of(model, result["usage"]["input_tokens"], result["usage"]["output_tokens"])
    SPEND["day"] += cost

    response = {**result, "cached": False, "cost_usd": round(cost, 6)}
    CACHE[key] = result
    return response

Run it: uvicorn app:app --port 8000, then test:

curl -s localhost:8000/ask -F [email protected] \
  -F question="What is the total amount and due date?" | python3 -m json.tool

Expected output:

{
  "answer": "The total amount is $4,318.20, due on November 14, 2026.",
  "sources": [1],
  "confidence": "high",
  "usage": {"input_tokens": 1243, "output_tokens": 61},
  "cached": false,
  "cost_usd": 0.001199
}

Ask again with the same file and question: cost_usd: 0.0, cached: true.

Production Checklist

Before this handles real traffic:

• Replace the in-memory cache with Redis (TTL of a few days) — a dict dies on every deploy.
• Move prices and budgets to config and alert when daily spend crosses 80%.
• Handle multi-page caps explicitly: for a 40-page PDF, either chunk it into parallel requests per page-range or route to a long-context model and send rendered pages — measure which is cheaper for your documents.
• Log citations and confidence and surface low-confidence answers in your UI ("I couldn't find this in the document") instead of fabricating.
• Rate-limit per API key so one noisy client can't eat the budget.
• Add retries with backoff around the model call for transient 429/5xx errors.

Cost Notes: Which Model Should You Route To?

For a typical 3-page document with images (~1,200–2,000 input tokens, ~60 output tokens per answer):

A practical pattern: route to the budget model first, and retry with a premium model only when confidence comes back "low". On most document mixes that cuts cost 5x–10x with almost no accuracy loss. See live pricing for all of these on Qubax models.

Conclusion

You now have a complete multimodal Q&A service: vision-based document reading, citation-grounded answers, JSON-structured output, semantic-free exact caching, and per-request cost metering with a hard budget cap. The same skeleton extends naturally — add Chroma/pgvector for cross-document search, add streaming for chat UX, or add tool calls so the model can query your database when the document isn't enough.

Full API reference for streaming, tool calling, and structured output lives at Qubax docs. Now go point it at your inbox.

FAQ

Do I need OCR software like Tesseract for this?

No. Vision-language models read rendered page images directly and generally beat classic OCR pipelines on real-world scans, handwriting, and complex layouts — OCR is only worth adding as a preprocessing step if you need exact text extraction for storage, not just answering.

Can this handle very large PDFs?

Render pages and cap what you send per request. For large documents, either process page ranges in parallel and merge, or use a long-context model that accepts all pages at once. Compare both approaches' token counts on your real documents to pick the cheaper one.

How accurate are the page citations?

High on clean documents with specific questions (totals, dates, names), lower on vague questions. Keep temperature=0, require JSON with sources, and show low-confidence answers to users with an honest "not found in document" message.

How much does a typical request cost?

Roughly $0.0002–$0.005 per answer depending on the model and document size. The exact-match cache reduces that to $0 for repeat questions, and budget caps make worst-case spend predictable.

Can I swap models without rewriting the code?

Yes — that's the point of the OpenAI-compatible format. Change the model parameter per request. Browsing Qubax models shows per-token prices so you can pick per workload.