Vision-language models (VLMs) have quietly become one of the most practical tools in the AI API toolbox. With a single API call, you can extract data from receipts, describe images for accessibility, moderate user-uploaded content, read screenshots, and answer questions about diagrams. In this tutorial, you'll build a production-ready image analysis endpoint in Python that works with any vision-capable model — plus the gotchas nobody warns you about (image encoding, size limits, and cost control).
What You'll Build
A small Python service that:
- Accepts an image (URL or uploaded file)
- Sends it to a vision model with a structured analysis prompt
- Returns structured JSON: description, detected objects, and extracted text (OCR)
- Handles errors, downsizes oversized images, and keeps costs predictable
Everything uses the OpenAI-compatible chat completions format, which means the same code works across providers — including vision models available through Qubax AI like Gemini 3.7 Flash, GPT-5.4, Qwen3 VL, or Claude Sonnet 5.
Prerequisites
- Python 3.10+
pip install openai pillow fastapi uvicorn python-multipart- An API key from your provider (store it in an environment variable, never in code)
Step 1: The Anatomy of a Vision API Call
Vision models accept image content parts alongside text. An image can be passed as a URL or as a base64-encoded data URI:
import base64
from openai import OpenAI
client = OpenAI(
api_key="YOUR_API_KEY",
base_url="https://api.qubax.ai/v1", # any OpenAI-compatible endpoint
)
MODEL = "gemini-3.7-flash" # any vision-capable model
def analyze_url(image_url: str, question: str) -> str:
resp = client.chat.completions.create(
model=MODEL,
messages=[{
"role": "user",
"content": [
{"type": "text", "text": question},
{"type": "image_url", "image_url": {"url": image_url}},
],
}],
max_tokens=1000,
)
return resp.choices[0].message.content
print(analyze_url(
"https://example.com/receipt.jpg",
"What is the total amount on this receipt?"
))The structure to remember: content becomes a list of parts instead of a string — one or more text parts plus one or more image_url parts. You can send multiple images in one request, which is great for comparing documents or frames from a video.
Step 2: Sending Local Files (Base64 Encoding)
Most real-world images aren't on a public URL — they're uploads. Encode them as data URIs:
from PIL import Image
import io
MAX_DIM = 1568 # beyond ~1.5–2MP, most VLMs downscale internally anyway
MAX_BYTES = 4 * 1024 * 1024
def prepare_image(file_bytes: bytes) -> str:
"""Resize if needed and return a base64 data URI."""
img = Image.open(io.BytesIO(file_bytes))
# Convert to RGB (handles PNG alpha, CMYK JPEGs, etc.)
if img.mode not in ("RGB", "L"):
img = img.convert("RGB")
# Downscale the longest edge if it's oversized
if max(img.size) > MAX_DIM:
img.thumbnail((MAX_DIM, MAX_DIM), Image.LANCZOS)
buf = io.BytesIO()
img.save(buf, format="JPEG", quality=85)
b64 = base64.b64encode(buf.getvalue()).decode()
return f"data:image/jpeg;base64,{b64}"Three important reasons for this step:
- Cost: many providers bill image inputs by resolution tiers. A 4000×3000 phone photo often costs more than a resized version while adding no accuracy.
- Limits: providers reject images above ~20MB after encoding.
- Reliability: weird color modes and giant EXIF payloads are a top source of 400 errors.
Step 3: Getting Structured JSON Output
Freeform descriptions are fine for demos, but production systems want parseable data. The robust pattern is to ask for JSON and validate it:
import json
from pydantic import BaseModel, ValidationError
class ImageAnalysis(BaseModel):
description: str
objects: list[str]
text_extracted: str
ANALYSIS_PROMPT = """Analyze the image and respond with ONLY a JSON object:
{
"description": "one-sentence description",
"objects": ["list of visible objects"],
"text_extracted": "all visible text, or empty string"
}"""
def analyze_image(image_data_uri: str) -> ImageAnalysis:
resp = client.chat.completions.create(
model=MODEL,
messages=[{
"role": "user",
"content": [
{"type": "text", "text": ANALYSIS_PROMPT},
{"type": "image_url", "image_url": {"url": image_data_uri}},
],
}],
response_format={"type": "json_object"}, # supported by many models
temperature=0,
max_tokens=800,
)
raw = resp.choices[0].message.content
try:
return ImageAnalysis.model_validate_json(raw)
except ValidationError:
# Model occasionally wraps JSON in markdown fences — strip and retry
cleaned = raw.strip().removeprefix("json").removeprefix("``").removesuffix("``") return ImageAnalysis.modelvalidatejson(cleaned)
Tips that matter:
- **`temperature=0`** for extraction tasks — you want determinism, not creativity.
- **`response_format={"type": "json_object"}`** is supported by many OpenAI-compatible models and reduces malformed output dramatically.
- **Validate anyway.** Models occasionally wrap JSON in code fences; a lenient cleanup path saves you pages of debugging.
## Step 4: Wrap It in a FastAPI Endpoint
python from fastapi import FastAPI, UploadFile, HTTPException
app = FastAPI()
@app.post("/analyze") async def analyze(file: UploadFile): if file.content_type not in ("image/jpeg", "image/png", "image/webp", "image/gif"): raise HTTPException(415, "Unsupported image type")
data = await file.read() if len(data) > MAX_BYTES: raise HTTPException(413, "Image too large")
try: result = analyzeimage(prepareimage(data)) except Exception as e: raise HTTPException(502, f"Vision model failed: {e}")
return result.model_dump()
Run it with `uvicorn main:app` and test:
bash curl -X POST http://localhost:8000/analyze \ -F "[email protected]"
## Step 5: Cost Control (Read This Before Deploying)
Vision requests are typically **much more expensive than text-only calls** because images consume significant input tokens. Protect yourself:
1. **Cap `max_tokens`** on output — runaway generations are the most common bill shock.
2. **Resize before sending.** A receipt doesn't need 12 megapixels; 1568px on the long edge is usually plenty.
3. **Skip the model when you can.** For pure barcode/QR reading, a dedicated library is free and perfect.
4. **Choose the right tier.** A lightweight VLM (Gemini 3.7 Flash, GPT-5.4 Mini, Qwen3 VL 30B) handles receipts, screenshots, and moderation at a fraction of frontier-model cost. Save the heavy models for dense charts and handwriting.
5. **Monitor usage per endpoint.** Tag requests so you can attribute spend.
Compare current vision-model pricing across providers on the [Qubax model catalog](https://qubax.ai/models) — the spread between providers for the same capability tier is often 5–10x.
## Common Pitfalls
- **Sending text and image in the wrong order** rarely matters, but *omitting* the text instruction while the model's default behavior guesses the task does. Always include an explicit instruction.
- **PDFs are not images.** Most chat-completions vision endpoints accept images only; rasterize PDF pages first (`pdf2image` + `Pillow`).
- **Low-quality OCR results?** Try increasing resolution slightly, cropping to the text region, or switching to a model known for strong OCR.
- **Rate limits hit harder with images** — large payloads take longer to upload and process. Add retries with exponential backoff on 429s.
## Where to Go From Here
Ideas that build directly on this service: multi-page document pipelines (rasterize → analyze → aggregate), an accessibility alt-text generator, a moderation queue for user uploads, or an agent that can "look" at screenshots as part of its tool loop. The code above is the foundation for all of them.
## FAQ
### Which models can analyze images via API?
All major vision-language models: Google's Gemini Flash/Pro lines, OpenAI's GPT-5.x family, Anthropic's Claude Sonnet and Opus, and open models like Qwen3 VL. They all use the same basic pattern — an image content part in the message. Browse vision-capable models on [Qubax AI](https://qubax.ai/models).
### How do I send a local image to an AI API?
Base64-encode it into a data URI (`data:image/jpeg;base64,...`) and pass it in an `image_url` content part. Resize large images first to control cost and avoid payload limits.
### How much does image analysis cost?
It varies by provider and image size — many providers bill images as token-equivalents or resolution tiers, so a large image can cost several times a text-only request. Resizing images and choosing lightweight vision models are the biggest levers; compare rates on the [Qubax catalog](https://qubax.ai/models).
### Can vision models extract text (OCR)?
Yes — modern VLMs handle printed text well and handwriting reasonably well. For best results, crop to the text region, use adequate resolution, set `temperature=0`, and consider models with dedicated OCR strengths for dense documents.
### Can I send multiple images in one request?
Yes. Add multiple `image_url` content parts to the same message — useful for comparing documents, diffing screenshots, or analyzing video frames.
## Step 6: Add Retry Logic and Streaming for Production
Two more pieces make this genuinely production-grade.
**Retries with backoff** — vision endpoints see transient 429s and 5xx more often than text endpoints because payloads are heavier:
python import time
def analyzewithretry(imagedatauri: str, retries: int = 3) -> ImageAnalysis: for attempt in range(retries): try: return analyzeimage(imagedata_uri) except Exception as e: if attempt == retries - 1: raise wait = 2 ** attempt # 1s, 2s, 4s print(f"Attempt {attempt + 1} failed ({e}); retrying in {wait}s") time.sleep(wait) ```
For anything user-facing, prefer exponential backoff with jitter and respect the Retry-After header when the provider sends one — hammering a rate-limited endpoint only extends your lockout.
Streaming the description — if you render results in a UI, streaming the text portion improves perceived latency massively. Most OpenAI-compatible APIs support stream=True; consume the deltas and let structured extraction run server-side in a second pass only when needed. A good pattern: stream a human-readable description to the screen while extracting the JSON quietly in the background — the user sees words in under a second, and your database still gets clean, validated records.
With retries, resizing, JSON validation, and cost caps in place, this ~100-line service handles real traffic. From here, plug it into a queue (Celery, RQ, or a serverless function) for batch workloads, and you have a complete vision microservice you can point at any provider's model with a one-line config change.
