How AI finds bugs in screenshots: UI analysis for QA

Manual UI review after every release eats time, and your eyes quickly stop noticing things. AI screenshot analysis takes over part of that work: an automated test captures the screen, the model looks at the image and returns a list of suspicious spots — typos, overlaps, missing buttons. Below is working Python code for Playwright, a prompt template and a way to calculate cost without guesswork.

Which models can read images

Claude and GPT models accept images. GLM 5.3 and GLM 5.3 Flash are text-only — sending them a screenshot makes no sense.

The model list changes; the current one is at GET https://api.zukko.pro/v1/models or on the "Models" page in your dashboard. Nobody can tell you in advance which model finds bugs best in your interface: run the same 10–20 screenshots with known defects through two or three models and compare how many defects each one caught and how many false positives it produced.

What the model notices and what it misses

The model works well as an additional layer of checks, but it does not replace pixel diffing (visual regression) or manual testing. It is reasonable to ask it to find:

• typos and inconsistent text in buttons, headings, hints;
• clearly overlapping or clipped elements;
• missing elements you listed in the prompt ("the login screen must have an email field, a password field and a Sign in button");
• hard-to-read text: too light on a light background, too small;
• text that contradicts the screen state ("Loading…" while data is already shown).

Anything that needs exact measurements works poorly: "15 px padding instead of 16", "the shade is 3% off the brand color". Use image comparison tools for that.

Send a screenshot with the Anthropic SDK

The screenshot is base64-encoded and passed as an image block. For the Anthropic SDK the base URL has no /v1 — the SDK adds it itself.

import base64
from anthropic import Anthropic

client = Anthropic(base_url="https://api.zukko.pro", api_key="fetch_your_key")

with open("screenshot.png", "rb") as f:
    image_data = base64.standard_b64encode(f.read()).decode("utf-8")

message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=2048,
    messages=[{
        "role": "user",
        "content": [
            {"type": "image", "source": {
                "type": "base64", "media_type": "image/png", "data": image_data}},
            {"type": "text", "text": "Find UI bugs in this screenshot. Reply with JSON."},
        ],
    }],
)
print(message.content[0].text)
print(message.usage.input_tokens, message.usage.output_tokens)

According to Anthropic's documentation, PNG, JPEG, GIF and WebP are supported, and the maximum size is 8000×8000 pixels. Large images are downscaled along the long edge before processing, so small text on a huge full-page screenshot may become unreadable.

The same request in OpenAI format

If your project already uses the OpenAI SDK, pass the image as a data URL in an image_url block. The base URL includes /v1:

import base64
from openai import OpenAI

client = OpenAI(base_url="https://api.zukko.pro/v1", api_key="fetch_your_key")

with open("screenshot.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode("utf-8")

resp = client.chat.completions.create(
    model="gpt-5.6-sol",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Find UI bugs in this screenshot. Reply with JSON."},
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
        ],
    }],
)
print(resp.choices[0].message.content)

The gateway converts between formats, but before running hundreds of screens, test your chosen model and format combination on a single screenshot.

QA prompt template

"Find bugs" gives a vague answer. List the criteria and the elements the screen must contain, and fix the response format strictly:

QA_PROMPT = """You are reviewing a web app screenshot before release.
Screen: {screen}. It must contain: {required}.

Report only real problems:
1. Typos and errors in UI text.
2. Overlapping, clipped or overflowing elements.
3. Missing required elements from the list above.
4. Hard-to-read text (low contrast, font too small).
5. Text that contradicts the screen state.

If unsure, leave the item out. Reply with JSON only, no comments:
{{"bugs": [{{"type": "typo|overlap|missing|contrast|state",
            "location": "where on screen",
            "description": "what is wrong",
            "severity": "critical|minor"}}],
  "status": "ok|warn|critical"}}"""

The double braces are there because the string is filled with .format(). Models sometimes wrap JSON in a code block, so parse the answer leniently:

import json
import re

def parse_json(text: str):
    match = re.search(r"\{.*\}", text, re.S)
    if not match:
        return None
    try:
        return json.loads(match.group(0))
    except json.JSONDecodeError:
        return None

Playwright integration

The script visits several pages, captures each one and collects the findings in a report. It uses the async client so it doesn't block Playwright's event loop.

import asyncio
import base64
import json
from anthropic import AsyncAnthropic
from playwright.async_api import async_playwright

SCREENS = [
    ("https://example.com/login", "login", "email field, password field, Sign in button"),
    ("https://example.com/settings", "settings", "Save button"),
]

async def main():
    client = AsyncAnthropic(base_url="https://api.zukko.pro", api_key="fetch_your_key")
    report = []

For mobile layouts, set a narrow viewport (viewport={"width": 375, "height": 812}) or use a device profile from p.devices. Capture long pages as several screens rather than one full_page=True shot, so text stays readable after downscaling.

How much AI screenshot analysis costs

The number of tokens per image depends on its size and the model, so don't rely on someone else's estimates — measure on your own screens. The API response contains usage.input_tokens (image + prompt) and usage.output_tokens. For the Anthropic format you can call POST /v1/messages/count_tokens with the same content beforehand.

Formula for claude-sonnet-5:

Example: if your screen came out at 2,000 input and 400 output tokens, then (2,000 × 0.3 + 400 × 0.3) / 1,000,000 = $0.00072 per screenshot, and a run of 300 screens is 300 × 0.00072 = $0.216. For gpt-5.6-luna, use 0.1 instead of 0.3. The script above already stores tokens in the report, so you can see the spend per screen.

To keep a dedicated QA key from eating the budget, give it a daily limit and a list of allowed models in the dashboard — see key limits↗. You can create a key on the API keys↗ page, and the request parameters for images are covered in the Messages API↗ reference.

FAQ

Can I send screenshots to GLM to save money?

No. GLM 5.3 and GLM 5.3 Flash do not accept images. The cheapest image-capable option is gpt-5.6-luna at $0.1 per 1M tokens; test on your own screens whether it is good enough for your tasks.

What about sensitive data in screenshots?

Run checks on test data and test accounts. If real data is unavoidable, mask the fields before capturing: Playwright's page.screenshot() has a mask parameter that takes a list of locators.

Why does the model report "bugs" that aren't there?

It happens, especially with a vague prompt. Three things help: a list of required screen elements, an "if unsure, leave it out" instruction, and a manual check of critical findings before filing a ticket.

Should I send the whole page or parts of it?

A regular 1280×800 screen — whole. A long page — several shots, one viewport height each; otherwise small text may become unreadable after downscaling.