Extracting structured data from documents

Why this is a vision task, not an OCR script

Plain OCR returns text without structure — you still need regex or manual work to map it onto fields, and that breaks on the first document with a different layout. The Claude and GPT models in the catalogue accept images (vision; GLM doesn't have it), so you can hand over a scan or a photo of a document directly and ask for a structured answer, skipping the intermediate text-recognition step.

Request format with an image

Through the Anthropic-compatible POST /v1/messages endpoint, an image is passed as a separate content block in base64:

import base64, requests

with open("invoice.jpg", "rb") as f:
    img_b64 = base64.b64encode(f.read()).decode()

resp = requests.post(
    "https://api.zukko.pro/v1/messages",
    headers={
        "Authorization": "Bearer fetch_your_key",
        "anthropic-version": "2023-06-01",
        "Content-Type": "application/json",
    },
    json={
        "model": "claude-opus-5-5",

For difficult scans — poor quality, handwritten notes, an unusual layout — try claude-opus-5-5 with extended thinking and check on your own documents whether it's less prone to inventing values where the text is unreadable.

A strict schema via tools instead of hoping for a format

Asking the model to "return JSON" works, but it might wrap it in an explanation or a markdown block. It's more reliable to define a schema through a tool description and force the model to answer with a tool call — then parsing is deterministic:

{
  "model": "claude-opus-5-5",
  "max_tokens": 1024,
  "tools": [{
    "name": "extract_invoice",
    "description": "Extracted invoice fields",
    "input_schema": {
      "type": "object",
      "properties": {
        "number": {"type": "string"},
        "date": {"type": "string"},
        "total": {"type": "number"},
        "vendor_tax_id": {"type": "string"}
      },
      "required": ["number", "date", "total"]
    }
  }],
  "messages": [ ... ]
}

The same idea works through the OpenAI-compatible POST /v1/chat/completions with tools in the body — if your pipeline is already written against the OpenAI SDK, there's no need to rewrite it for the Anthropic format.

Multi-page PDFs

The /v1/messages endpoint accepts images, not a whole PDF, so a multi-page document needs to be split into page images first (for example with pdftoppm or a library like pdf2image) and passed as several image blocks in one message — the model sees all pages at once and can cross-reference data between them, say a total on the last page against a line-item list on the first.

from pdf2image import convert_from_path

pages = convert_from_path("contract.pdf", dpi=200)
content = []
for page in pages:
    buf = io.BytesIO()
    page.save(buf, format="JPEG")
    content.append({
        "type": "image",
        "source": {"type": "base64", "media_type": "image/jpeg", "data": base64.b64encode(buf.getvalue()).decode()},
    })
content.append({"type": "text", "text": "Extract the contract parties, amount, and signing date. Return JSON."})

A confidence field instead of a silent mistake

For fields the model couldn't read confidently — a smudged stamp, a cropped scan edge — it helps to explicitly ask for null instead of a guessed value, plus a separate boolean needs_review field. That turns a silent mistake into an explicit signal for the manual review queue, instead of a plausible-looking but wrong value quietly landing in the database.

Batch processing a large archive

Before sending an archive of several thousand documents through the pipeline, estimate the cost: an image in a request takes up a noticeable share of the context, and POST /v1/messages/count_tokens will size a single sample exactly, for free — multiply by the document count and check against model prices from GET /v1/models/info.

Errors specific to batch processing

On a large archive, at least one of the typical errors will show up: 403 model_not_allowed_for_key, if the key has a model allowlist set in the dashboard and the script points at the wrong one; 429 with key_daily_limit_reached, if the key's daily limit is lower than what the whole archive needs in one day; 503 model_disabled, if a model is temporarily disabled in the catalogue — worth building in a fallback to another vision model from the current GET https://api.zukko.pro/v1/models list (Claude or GPT — GLM has no vision) rather than stopping the whole pipeline.

Date formats across documents

Dates on invoices and receipts show up in different formats — DD.MM.YYYY, MM/DD/YYYY, YYYY-MM-DD — and without explicit instructions a model sometimes defaults to its own internal locale's format. It helps to specify directly in the prompt which format to return the date in, for example always ISO 8601, so you don't spend time post-processing mismatched strings coming from different documents in the same batch.

File naming and traceability

When processing an archive, it's worth saving the original filename and page number directly in the JSON response, not just in the filename on disk. Then, when tracking down a mismatch, it's clear where a specific record came from, even after files get renamed, moved to a different folder, or merged into one shared CSV with the results of the whole batch.

Checking the output

A good test before running the pipeline on the full archive is to run 20-30 documents whose values you already know and compare them line by line. Models sometimes confuse similar fields (a document date with a payment date, a vendor's tax ID with a buyer's), and a sample catches that immediately rather than after the mistake lands in a database across thousands of records.