How to Extract Structured Data From Images With Vision AI APIs: Complete Tutorial

Every company has a folder somewhere full of PDFs and photos — receipts, invoices, delivery manifests, warranty cards — that someone has to retype into a spreadsheet. Traditional OCR (Tesseract, cloud OCR services) gets you raw text, but not structure: no field labels, no type coercion, no idea that "10/08/2026" in one document and "August 10, 2026" in another mean the same thing.

Vision-language models changed this. You send the image, describe the JSON you want, and get structured data back in one call. No template rules, no regex graveyards, no coordinate math.

In this tutorial, you'll build a production-shaped extractor in Python that:

1. Sends an image to a vision-capable model via the OpenAI-compatible chat completions API
2. Forces a strict JSON schema using response_format
3. Validates the result with Pydantic
4. Retries intelligently on failure
5. Tracks token usage so you know your cost per document

The Pattern in One Paragraph

You embed the image as a base64 data URL in the message content, pair it with a prompt that specifies the schema, and request a JSON response. The model reads the image natively — no OCR step, no layout analysis, no bounding boxes. Vision models handle rotated phone photos, cramped tables, handwriting, and multi-column layouts that make classic OCR weep.

Prerequisites

• Python 3.10+
• An API key from qubax.ai (one key works across all models below)
• Packages: openai, pydantic, httpx (install: pip install openai pydantic httpx)

The same code works with any OpenAI-compatible endpoint. We'll use Qubax so we can swap models without changing anything but a string.

Step 1: Define Your Schema First

Start with the shape of the data you want, not the prompt. Pydantic model:

from pydantic import BaseModel, Field
from typing import Optional
from decimal import Decimal
from enum import Enum

class LineItem(BaseModel):
    description: str
    quantity: float
    unit_price: Decimal
    total: Decimal

class PaymentMethod(str, Enum):
    CARD = "card"
    CASH = "cash"
    TRANSFER = "transfer"
    UNKNOWN = "unknown"

class Receipt(BaseModel):
    merchant: str
    date: str  # ISO 8601, normalized
    currency: str = Field(pattern=r"^[A-Z]{3}$")
    total: Decimal
    tax: Optional[Decimal] = None
    payment_method: PaymentMethod = PaymentMethod.UNKNOWN
    line_items: list[LineItem]

Why this matters: the schema is the prompt. A model asked for {"date": "ISO 8601"} returns normalized dates far more reliably than one asked to "extract the date." Typed constraints (enum, pattern) give you validation for free at the Pydantic layer.

Step 2: Encode the Image

Vision models accept image URLs or base64 data URLs. For local files, base64:

import base64
from pathlib import Path

def encode_image(path: str) -> str:
    data = Path(path).read_bytes()
    b64 = base64.b64encode(data).decode()
    return f"data:image/jpeg;base64,{b64}"

Keep source images under ~1 MB where you can. Gigantic scans cost tokens and add nothing — the model downsamples internally anyway. If you're processing scans at 300 DPI, downscale to ~150 DPI first; accuracy is unaffected for text extraction and you'll cut image tokens substantially.

Step 3: The Extraction Call

import json
from openai import OpenAI

client = OpenAI(
    api_key="YOUR_QUBAX_KEY",
    base_url="https://api.qubax.ai/v1",
)

SYSTEM_PROMPT = """You are a precise data-extraction engine.
Extract fields from the receipt image exactly as shown.
Rules:
- Dates: always ISO 8601 (YYYY-MM-DD).
- Currency: ISO 4217 three-letter code.
- If a field is unreadable or absent, use null. Never guess.
- Numbers: plain numbers, no currency symbols or thousands separators.
Return JSON only."""

def extract_receipt(image_path: str, model: str = "zhipuai/glm-5v-turbo") -> Receipt:
    response = client.chat.completions.create(
        model=model,
        temperature=0,
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": f"Extract a Receipt. Schema: {Receipt.model_json_schema()}"},
                    {"type": "image_url", "image_url": {"url": encode_image(image_path)}},
                ],
            },
        ],
        # vision tokens are billed as input tokens
    )
    return Receipt.model_validate_json(response.choices[0].message.content)

Two details doing heavy lifting:

• `temperature=0`: extraction has one right answer; remove sampling variance. (Not sure why? Read our temperature explainer.)
• `response_format={"type": "json_object"}`: constrains decoding to valid JSON at the API layer — no markdown fences to strip, no truncation mid-object.

Step 4: Validate and Retry

LLMs occasionally return schema-valid-JSON that fails Pydantic (string where a Decimal belongs, invalid enum). Retry with the validation error as feedback:

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(min=1, max=8))
def extract_with_retry(image_path: str, model: str) -> Receipt:
    try:
        return extract_receipt(image_path, model)
    except Exception as e:
        err = str(e)[:500]
        raise RuntimeError(
            f"Validation failed: {err}. Fix the JSON to match the schema exactly."
        ) from e

For a cleaner pattern, catch the validation error and re-send the conversation including the model's bad output plus the error — the model usually self-corrects in one round. In practice, a two-attempt budget fixes the vast majority of malformed extractions.

Step 5: Handle "Not a Receipt" Gracefully

The worst failure mode isn't a bad field — it's confidently extracting a "receipt" from a photo of a cat. Add a discriminator:

class ExtractionResult(BaseModel):
    is_receipt: bool
    receipt: Optional[Receipt] = None

# prompt addition: 'If the image is not a receipt, set is_receipt=false and receipt=null.'

Then branch on is_receipt in your pipeline. This costs nothing in accuracy and saves you from garbage rows downstream. It also gives you a clean signal for routing: non-receipts can go to a cheaper classifier before ever hitting the big model.

Step 6: Pick the Right Model (and Know the Cost)

Vision capability and price vary wildly. At Qubax prices (per 1M tokens, input/output):

A receipt at ~1,000 image tokens + 300 prompt tokens and ~200 output tokens costs roughly $0.0004 with GLM 5V Turbo — about 4 cents per 100 receipts.

That's the quiet revolution here: document AI that used to require a contract with an enterprise OCR vendor now costs fractions of a cent per page and runs through the same API as your chat models. At even modest volumes, model choice is your biggest cost lever: the gap between running everything on a flagship vision model versus a mid-tier one is often 5–10x on the monthly bill for a modest accuracy delta.

A pragmatic production pattern is tiered routing: run everything through the cheap model first, score confidence, and escalate only ambiguous or failed extractions to the premium model. If 85% of your documents are clean, tiered routing cuts spend dramatically while keeping accuracy at flagship levels where it matters.

Step 7: Track Tokens (So Billing Never Surprises You)

The response includes usage. Log it:

usage = response.usage
print(f"in={usage.prompt_tokens} out={usage.completion_tokens} total={usage.total_tokens}")

Multiply by your model's price per million tokens and you have exact per-document cost. Build this logging in from day one — "why is our invoice so high?" is the most common question we hear from teams scaling AI pipelines.

Common Pitfalls

1. No `response_format`. You'll spend your life stripping markdown fences and debugging truncated JSON. Turn it on.
2. Guessing instead of nulling. Without an explicit "never guess" rule, models fabricate plausible values for unreadable fields — the most dangerous failure in financial data.
3. One giant schema for everything. If documents vary (receipts vs invoices vs delivery slips), classify first, then extract with a per-type schema. Accuracy jumps and prompts stay debuggable.
4. Forgetting images count as input tokens. A 2 MB photo can be thousands of tokens. Downscale before sending.
5. Testing only on clean scans. Your pipeline will meet crumpled receipts in the wild. Build a test set of ugly real-world photos.
6. Ignoring rotation and EXIF. Most vision models handle EXIF rotation, but strips from scanner pipelines sometimes don't. Normalize orientation before encoding.

What We Built

A ~100-line extractor that turns arbitrary receipt photos into validated Python objects, with retries, cost tracking, and model portability across any OpenAI-compatible endpoint. The same skeleton works for invoices, ID documents, forms, whiteboard photos, product labels — anywhere messy reality meets structured data.

Try it with your own documents:

• Browse vision-capable models and pricing: qubax.ai/models
• Full API reference: qubax.ai/docs

FAQ

Do I need OCR if I use a vision AI model?

No. Vision-language models read the image directly — including layout, tables, and handwriting — so a separate OCR step is unnecessary for most extraction tasks. OCR still wins for bit-perfect archival transcription of plain text.

Which vision model should I start with?

Start with GLM 5V Turbo — it's fast, inexpensive, and strong on standard documents. Escalate to GPT-5.6 Sol or Qwen3 VL 235B for dense tables or handwriting. All are available at qubax.ai/models.

How much does image extraction cost?

Typically $0.0002–$0.001 per document depending on image size and model. A receipt-scale image on GLM 5V Turbo runs ~$0.0004; the same on a flagship model might run 3–5x more. Track usage on every call to know exactly.

How do I force the model to return valid JSON?

Use response_format={"type": "json_object"} plus a schema in the prompt, then validate with Pydantic and retry on failure with the error message included. This three-layer approach (constrain, validate, retry) gets you to production reliability.

Can vision models handle handwriting?

Yes, reasonably well — modern vision models substantially outperform traditional OCR on handwriting. For messy handwriting, use the strongest model you can justify and consider a two-pass approach: extract, then a second call to verify ambiguous fields against the image.

Is my document data sent anywhere I should worry about?

Data handling depends on your provider's retention and privacy policy. Review the terms for any API you send sensitive documents to — especially for regulated data. Qubax's data policies are documented at qubax.ai/docs.

Does this work with PDFs?

Most vision APIs accept images, not PDFs, natively. Convert pages to images first (pdftoppm or pdf2image), then send as shown. For multi-page PDFs, loop pages and merge with a final normalization pass.