Recognize a receipt from a screenshot with AI

Receipt recognition is a common task in expense tracking, accounting services, and expense-report apps. A user photographs a receipt or sends a screenshot from a banking app, and the system has to extract the date, total, and line items. Below: how to send an image to a model via API in OpenAI format, get JSON back, and validate it before saving.

Why an AI model, not just OCR

Classic OCR (for example, Tesseract) returns text but doesn't understand structure: which line is the total, which is a discount, where the store name is. A model with image support reads the whole receipt and returns fields in the format you need: it sees the "Total" line and knows the number next to it is the amount.

Pros:

• Structured data (JSON) right away, without a separate parser.
• Works with different formats: register receipts, order screenshots, PDF receipts saved as images.
• Extraction rules change by editing the prompt, not the code.

Cons:

• Every request costs money, while local OCR is free.
• The model can misread a digit, so validation is required (below).
• The image goes to an external service — keep that in mind for personal data.

Sending an image in OpenAI format

In the OpenAI-compatible API, an image is passed as part of the message content: an element {"type": "image_url", "image_url": {"url": ...}}, where url is a data URL like data:image/jpeg;base64,<data>. A text part with the instruction goes next to it.

Shrink the photo before sending: for Claude models, the image cost depends on its pixel size, and large images are downscaled on the model side anyway.

import base64
import io
import json
from openai import OpenAI
from PIL import Image

client = OpenAI(
    base_url="https://api.zukko.pro/v1",
    api_key="fetch_your_key",
)

SYSTEM_PROMPT = """You extract data from receipt images.
Return ONLY JSON, no markdown:
{

If you already have PNG or WebP bytes and don't need to resize them, just change the type in the data URL: data:image/png;base64,....

If the receipt is behind a link

The OpenAI format also accepts a plain https link in image_url.url, but then the model provider has to download the image: the link must be public and reachable from outside. It's more reliable to download the file yourself and send the same data URL:

import tempfile
import requests

def recognize_receipt_from_url(image_url: str) -> dict:
    resp = requests.get(image_url, timeout=30)
    resp.raise_for_status()
    with tempfile.NamedTemporaryFile(suffix=".img") as tmp:
        tmp.write(resp.content)
        tmp.flush()
        return recognize_receipt(prepare_image(tmp.name))

This also works with private storage (S3 with a signed link, an internal file server) that an external service can't reach.

Validate the extracted data

The model can misread a digit or treat a discount as an item. Check formats and compare the item sum with the total:

from datetime import date

def validate_receipt(data: dict, tolerance: float = 1.0) -> dict:
    """Check fields; on mismatches the receipt goes to manual review"""
    errors = []

    if data.get("date"):
        try:
            date.fromisoformat(data["date"])
        except (ValueError, TypeError):
            errors.append(f"invalid date: {data['date']}")

    total = data.get("total")
    if total is not None and (not isinstance(total, (int, float)) or total < 0):

Show receipts flagged needs_review to the user for confirmation instead of saving them silently.

Saving to a database

import sqlite3
from datetime import datetime

def save_receipt(data: dict, db_path: str = "receipts.db") -> int | None:
    check = validate_receipt(data)
    if not check["valid"]:
        print("Receipt not saved:", check["errors"])
        return None
    conn = sqlite3.connect(db_path)
    conn.execute("""CREATE TABLE IF NOT EXISTS receipts (
        id INTEGER PRIMARY KEY, date TEXT, total REAL, currency TEXT,
        vendor TEXT, confidence REAL, created_at TEXT)""")
    conn.execute("""CREATE TABLE IF NOT EXISTS receipt_items (
        id INTEGER PRIMARY KEY, receipt_id INTEGER REFERENCES receipts(id),

Image size

• According to Anthropic's documentation, an image costs roughly ⌈width/28⌉ × ⌈height/28⌉ tokens on Claude models, and newer models accept up to 2576 px on the long edge and up to ~4,800 tokens per image. A 1176×1568 photo after prepare_image is about 2,350 tokens.
• For receipts, 1200–1600 px on the long edge is usually enough: the text stays legible and the request is cheaper.
• The gateway rejects an oversized request with a 413 error — resize and compress photos before sending.

Model choice and cost

The example uses claude-sonnet-5 ($0.30/$0.30 per 1M tokens); other Claude models accept images too, for example claude-opus-5-5 at the same price. To check whether a specific model in the catalog supports images, send a short test request with a picture. Which model is more accurate on your receipts will show up in a comparison on 30–50 real photos with reference values.

Estimate for 1,000 receipts per month: ~2,400 tokens per image + ~300 for the prompt, ~300 output tokens (JSON):

If you send phone photos without resizing, an image can cost up to ~4,800 tokens: (1000 × 5100 × 0.30 / 1M) + $0.09 = $1.62 per month.

Current model list: GET https://api.zukko.pro/v1/models or the Models page in your dashboard.

Personal data

A receipt may include the cashier's name, part of a card number, the store address, and sometimes the buyer's full name. Recommendations:

1. Don't keep source images longer than necessary and don't write them to logs.
2. Store only the fields you need (date, total, items).
3. If your rules don't allow sending this data to an external service, use local OCR.

Set up the key

Create a key in the Zukko dashboard↗, restrict it to the model you need, and set a daily limit. For 30 receipts a day (about 900 a month), $1 a month covers the estimate above. The request format is described in the Chat Completions reference↗, basic setup in the Python examples↗.

FAQ

Can I use a cheaper model?

You can try, if the model accepts images: send a test request with a picture and run the same sample of 30–50 receipts. Compare not just the price but also the share of receipts sent to manual review.

What if the receipt has handwritten text?

The model will try to read it, but results are less predictable. Such receipts are almost always worth sending to the user for confirmation.

How do I process many receipts?

Send requests in parallel in small groups and handle a 429 error by retrying after a pause. The key's daily limit protects your budget if something goes wrong.

Do I need to fine-tune on receipts from a specific store?

No. If a format is recognized poorly, describe that format in the system prompt or add a sample answer, and check the photo quality.