How to Build an AI Image Analysis API in Python: Vision Models, JSON Output, and Cost Control

Vision-language models (VLMs) have quietly become one of the most practical tools in the AI API toolbox. With a single API call, you can extract data from receipts, describe images for accessibility, moderate user-uploaded content, read screenshots, and answer questions about diagrams. In this tutorial, you'll build a production-ready image analysis endpoint in Python that works with any vision-capable model — plus the gotchas nobody warns you about (image encoding, size limits, and cost control).

What You'll Build

A small Python service that:

1. Accepts an image (URL or uploaded file)
2. Sends it to a vision model with a structured analysis prompt
3. Returns structured JSON: description, detected objects, and extracted text (OCR)
4. Handles errors, downsizes oversized images, and keeps costs predictable

Everything uses the OpenAI-compatible chat completions format, which means the same code works across providers — including vision models available through Qubax AI like Gemini 3.7 Flash, GPT-5.4, Qwen3 VL, or Claude Sonnet 5.

Prerequisites

• Python 3.10+
• pip install openai pillow fastapi uvicorn python-multipart
• An API key from your provider (store it in an environment variable, never in code)

Step 1: The Anatomy of a Vision API Call

Vision models accept image content parts alongside text. An image can be passed as a URL or as a base64-encoded data URI:

import base64
from openai import OpenAI

client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.qubax.ai/v1",  # any OpenAI-compatible endpoint
)

MODEL = "gemini-3.7-flash"  # any vision-capable model

def analyze_url(image_url: str, question: str) -> str:
    resp = client.chat.completions.create(
        model=MODEL,
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": question},
                {"type": "image_url", "image_url": {"url": image_url}},
            ],
        }],
        max_tokens=1000,
    )
    return resp.choices[0].message.content

print(analyze_url(
    "https://example.com/receipt.jpg",
    "What is the total amount on this receipt?"
))

The structure to remember: content becomes a list of parts instead of a string — one or more text parts plus one or more image_url parts. You can send multiple images in one request, which is great for comparing documents or frames from a video.

Step 2: Sending Local Files (Base64 Encoding)

Most real-world images aren't on a public URL — they're uploads. Encode them as data URIs:

from PIL import Image
import io

MAX_DIM = 1568   # beyond ~1.5–2MP, most VLMs downscale internally anyway
MAX_BYTES = 4 * 1024 * 1024

def prepare_image(file_bytes: bytes) -> str:
    """Resize if needed and return a base64 data URI."""
    img = Image.open(io.BytesIO(file_bytes))

    # Convert to RGB (handles PNG alpha, CMYK JPEGs, etc.)
    if img.mode not in ("RGB", "L"):
        img = img.convert("RGB")

    # Downscale the longest edge if it's oversized
    if max(img.size) > MAX_DIM:
        img.thumbnail((MAX_DIM, MAX_DIM), Image.LANCZOS)

    buf = io.BytesIO()
    img.save(buf, format="JPEG", quality=85)
    b64 = base64.b64encode(buf.getvalue()).decode()
    return f"data:image/jpeg;base64,{b64}"

Three important reasons for this step:

• Cost: many providers bill image inputs by resolution tiers. A 4000×3000 phone photo often costs more than a resized version while adding no accuracy.
• Limits: providers reject images above ~20MB after encoding.
• Reliability: weird color modes and giant EXIF payloads are a top source of 400 errors.

Step 3: Getting Structured JSON Output

Freeform descriptions are fine for demos, but production systems want parseable data. The robust pattern is to ask for JSON and validate it:

import json
from pydantic import BaseModel, ValidationError

class ImageAnalysis(BaseModel):
    description: str
    objects: list[str]
    text_extracted: str

ANALYSIS_PROMPT = """Analyze the image and respond with ONLY a JSON object:
{
  "description": "one-sentence description",
  "objects": ["list of visible objects"],
  "text_extracted": "all visible text, or empty string"
}"""

def analyze_image(image_data_uri: str) -> ImageAnalysis:
    resp = client.chat.completions.create(
        model=MODEL,
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": ANALYSIS_PROMPT},
                {"type": "image_url", "image_url": {"url": image_data_uri}},
            ],
        }],
        response_format={"type": "json_object"},  # supported by many models
        temperature=0,
        max_tokens=800,
    )
    raw = resp.choices[0].message.content
    try:
        return ImageAnalysis.model_validate_json(raw)
    except ValidationError:
        # Model occasionally wraps JSON in markdown fences — strip and retry
        cleaned = raw.strip().removeprefix("```json").removeprefix("```").removesuffix("```")
        return ImageAnalysis.model_validate_json(cleaned)

Tips that matter:

• `temperature=0` for extraction tasks — you want determinism, not creativity.
• `response_format={"type": "json_object"}` is supported by many OpenAI-compatible models and reduces malformed output dramatically.
• Validate anyway. Models occasionally wrap JSON in code fences; a lenient cleanup path saves you pages of debugging.

Step 4: Wrap It in a FastAPI Endpoint

from fastapi import FastAPI, UploadFile, HTTPException

app = FastAPI()

@app.post("/analyze")
async def analyze(file: UploadFile):
    if file.content_type not in ("image/jpeg", "image/png", "image/webp", "image/gif"):
        raise HTTPException(415, "Unsupported image type")

    data = await file.read()
    if len(data) > MAX_BYTES:
        raise HTTPException(413, "Image too large")

    try:
        result = analyze_image(prepare_image(data))
    except Exception as e:
        raise HTTPException(502, f"Vision model failed: {e}")

    return result.model_dump()

Run it with uvicorn main:app and test:

curl -X POST http://localhost:8000/analyze \
  -F "[email protected]"

Step 5: Cost Control (Read This Before Deploying)

Vision requests are typically much more expensive than text-only calls because images consume significant input tokens. Protect yourself:

1. Cap `max_tokens` on output — runaway generations are the most common bill shock.
2. Resize before sending. A receipt doesn't need 12 megapixels; 1568px on the long edge is usually plenty.
3. Skip the model when you can. For pure barcode/QR reading, a dedicated library is free and perfect.
4. Choose the right tier. A lightweight VLM (Gemini 3.7 Flash, GPT-5.4 Mini, Qwen3 VL 30B) handles receipts, screenshots, and moderation at a fraction of frontier-model cost. Save the heavy models for dense charts and handwriting.
5. Monitor usage per endpoint. Tag requests so you can attribute spend.

Compare current vision-model pricing across providers on the Qubax model catalog — the spread between providers for the same capability tier is often 5–10x.

Common Pitfalls

• Sending text and image in the wrong order rarely matters, but omitting the text instruction while the model's default behavior guesses the task does. Always include an explicit instruction.
• PDFs are not images. Most chat-completions vision endpoints accept images only; rasterize PDF pages first (pdf2image + Pillow).
• Low-quality OCR results? Try increasing resolution slightly, cropping to the text region, or switching to a model known for strong OCR.
• Rate limits hit harder with images — large payloads take longer to upload and process. Add retries with exponential backoff on 429s.

Where to Go From Here

Ideas that build directly on this service: multi-page document pipelines (rasterize → analyze → aggregate), an accessibility alt-text generator, a moderation queue for user uploads, or an agent that can "look" at screenshots as part of its tool loop. The code above is the foundation for all of them.

FAQ

Which models can analyze images via API?

All major vision-language models: Google's Gemini Flash/Pro lines, OpenAI's GPT-5.x family, Anthropic's Claude Sonnet and Opus, and open models like Qwen3 VL. They all use the same basic pattern — an image content part in the message. Browse vision-capable models on Qubax AI.

How do I send a local image to an AI API?

Base64-encode it into a data URI (data:image/jpeg;base64,...) and pass it in an image_url content part. Resize large images first to control cost and avoid payload limits.

How much does image analysis cost?

It varies by provider and image size — many providers bill images as token-equivalents or resolution tiers, so a large image can cost several times a text-only request. Resizing images and choosing lightweight vision models are the biggest levers; compare rates on the Qubax catalog.

Can vision models extract text (OCR)?

Yes — modern VLMs handle printed text well and handwriting reasonably well. For best results, crop to the text region, use adequate resolution, set temperature=0, and consider models with dedicated OCR strengths for dense documents.

Can I send multiple images in one request?

Yes. Add multiple image_url content parts to the same message — useful for comparing documents, diffing screenshots, or analyzing video frames.

Step 6: Add Retry Logic and Streaming for Production

Two more pieces make this genuinely production-grade.

Retries with backoff — vision endpoints see transient 429s and 5xx more often than text endpoints because payloads are heavier:

import time

def analyze_with_retry(image_data_uri: str, retries: int = 3) -> ImageAnalysis:
    for attempt in range(retries):
        try:
            return analyze_image(image_data_uri)
        except Exception as e:
            if attempt == retries - 1:
                raise
            wait = 2 ** attempt  # 1s, 2s, 4s
            print(f"Attempt {attempt + 1} failed ({e}); retrying in {wait}s")
            time.sleep(wait)

For anything user-facing, prefer exponential backoff with jitter and respect the Retry-After header when the provider sends one — hammering a rate-limited endpoint only extends your lockout.

Streaming the description — if you render results in a UI, streaming the text portion improves perceived latency massively. Most OpenAI-compatible APIs support stream=True; consume the deltas and let structured extraction run server-side in a second pass only when needed. A good pattern: stream a human-readable description to the screen while extracting the JSON quietly in the background — the user sees words in under a second, and your database still gets clean, validated records.

With retries, resizing, JSON validation, and cost caps in place, this ~100-line service handles real traffic. From here, plug it into a queue (Celery, RQ, or a serverless function) for batch workloads, and you have a complete vision microservice you can point at any provider's model with a one-line config change.