GPT-5.6 Luna led UI grounding
It hit all 24 targets and returned the highest mean box IoU at 0.771. It also had the lowest reported grounding cost in this run at $0.0057.
Repeato research · LLM Vision benchmark
Can Multimodal LLMs Be Trusted with UI Testing?
A screenshot-only benchmark for UI test automation: locate actionable elements, recognize when an element is absent, detect visible layout defects, and record the operational cost of getting an answer.
Pilot edition2026-08-13
Six deterministic screens. Twenty-four grounding targets. 9 API models. A focused pilot, not a ranking.
Screens6
Grounding targets24
Defect categories6
Models9
Pilot findings
What this edition shows
These findings describe one retained response per case on a small synthetic corpus. They are useful signals for the next edition, not general model rankings or stability claims.
Element presence was nearly saturated
8 of 9 models reached 100% balanced accuracy. The track works as an absent-element sanity check, but this pilot version does little to separate most models.
Grok 4.5 led defect detection
It was the only model with 100% balanced accuracy on clean and defective layouts. Its defect-category accuracy was 67%, showing that noticing a problem and naming it precisely are different tasks.
What this means for test automation
Use the right kind of vision for each test step
Exact interaction, continuous observation, and semantic interpretation have different performance requirements. One vision technique should not be expected to optimize all three.
Local observationUp to 20 times per secondLLM requestOften 2–10 secondsHow we combine local vision and LLMsCollapse analysis
Repeato's local computer-vision loop can inspect the screen at 20 Hz—once every 50 ms. A vision-language-model request commonly needs 2–10 seconds for image transfer, inference, and the network round trip. That difference matters when a test must notice an element appearing, wait for movement to stop, or react before a transient state disappears.
Fast feedback for repeated interaction
Local visual fingerprints can continuously search for recorded targets and act as soon as the screen is ready. Repeating remote model calls at the same frequency would be impractical.
Predictable, private, and inexpensive
Local matching has no network round trip, provider availability, or per-request API charge. Screens remain on the workstation, and the same algorithm can run offline.
AI for questions matching cannot answer
LLM vision earns its latency when a test needs semantic judgment: interpreting text or numbers, understanding dynamic content, or checking concepts such as an upward-trending graph.
Practical split
Use local computer vision for the high-frequency control loop—finding targets, waiting, interaction, and visual comparison. Escalate selectively to AI vision when the assertion genuinely requires language or semantic understanding.
Experiments
Three distinct UI vision tasks
Grounding, element presence, and UI integrity answer different questions and use separate metrics. The pilot does not combine them into an overall score.
Experiment 1 · UI grounding
Can a model locate the requested UI element?
Each model received 24 actionable targets across six clean pilot screens. A schema-valid point inside the target counts as a hit; box IoU measures localization precision. Expand a row for scenario-level results.
- Holo3 122B A10B0.478
- Grok 4.50.445
- GPT-5.40.528
- Gemini 3.6 Flash0.645
- GPT-5.6 Sol0.677
- GPT-5.6 Luna0.771
- Claude Sonnet 50.043
- Claude Opus 50.314
- Gemini 3.1 Pro Preview0.632
- Holo3 122B A10B$0.0126
- Grok 4.5$0.0669
- GPT-5.4$0.1196
- Gemini 3.6 Flash$0.0637
- GPT-5.6 Sol$0.2453
- GPT-5.6 Luna$0.0057
- Claude Sonnet 5$0.0990
- Claude Opus 5$0.2474
- Gemini 3.1 Pro Preview$0.0726
- Holo3 122B A10B1.68 s
- Grok 4.51.67 s
- GPT-5.42.99 s
- Gemini 3.6 Flash1.83 s
- GPT-5.6 Sol3.23 s
- GPT-5.6 Luna3.55 s
- Claude Sonnet 53.91 s
- Claude Opus 54.24 s
- Gemini 3.1 Pro Preview11.89 s
Accuracy, cost, and latency trade-offHigher and further left is better. Color shows mean response latency.
1.67 s fast11.89 s slow
Methodology
How UI grounding is scored
Four targets per screenshot across common actionable UI-element types.
Point accuracy
Predicted click point falls inside the target box.
Box accuracy
Mean IoU and share of predictions with IoU ≥ 0.5.
Validity
Schema-valid response rate; one run means no stability score.
Experiment 2 · Element presence
Can a model tell when an element is not there?
A balanced task checks whether models reject plausible absent elements instead of confidently inventing them.
| 100% valid12/12 cases · 0 failed attempts | 100% | 100% | 100% | 0% | 1.29 s | $0.3636per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid12/12 cases · 0 failed attempts | 100% | 100% | 100% | 0% | 7.45 s | $0.1414per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid12/12 cases · 0 failed attempts | 100% | 100% | 100% | 0% | 7.55 s | $3.2402per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid12/12 cases · 0 failed attempts | 100% | 100% | 100% | 0% | 7.82 s | $2.3426per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid12/12 cases · 0 failed attempts | 100% | 100% | 100% | 0% | 9.85 s | $7.4788per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid12/12 cases · 0 failed attempts | 100% | 100% | 100% | 0% | 7.50 s | $2.2346per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid12/12 cases · 0 failed attempts | 100% | 100% | 100% | 0% | 9.27 s | $2.9915per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid12/12 cases · 0 failed attempts | 100% | 100% | 100% | 0% | 7.79 s | $7.0680per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid12/12 cases · 0 failed attempts | 92% | 100% | 83% | 17% | 15.01 s | $2.3602per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
One pass per case; no confidence intervals or stability claims. Aggregate cost is extrapolated to 1,000 retained cases so incomplete runs do not appear artificially cheap. Rows with fewer than 12 responses still have a smaller case mix and are not directly comparable to complete rows.
Methodology
How element presence is scored
One known-present and one plausible-but-absent query per clean screen.
Detection
Balanced accuracy weights present and absent cases equally.
Class behavior
Recall, specificity, and false-positive rate expose error direction.
Boolean only
found=true or found=false; no localization score.
Experiment 3 · UI integrity
Can a model distinguish clean and broken layouts?
Paired clean controls and seeded defects test detection, defect classification, and localization without mixing those results with grounding.
Defect accuracy, cost, and latency trade-offHigher and further left is better. Color shows mean response latency.
2.44 s fast21.75 s slow
| 67% valid12/12 cases · 0 failed attempts | 58% | 17% | 100% | 0% | 2.44 s | $0.8251per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 92% valid12/12 cases · 0 failed attempts | 83% | 67% | 100% | 50% | 8.36 s | $0.2389per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid12/12 cases · 0 failed attempts | 92% | 83% | 100% | 67% | 9.02 s | $5.2075per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 75% valid12/12 cases · 0 failed attempts | 75% | 83% | 67% | 67% | 8.43 s | $3.8341per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 83% valid12/12 cases · 0 failed attempts | 83% | 67% | 100% | 67% | 10.15 s | $11.5029per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid12/12 cases · 0 failed attempts | 100% | 100% | 100% | 67% | 7.40 s | $2.7465per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 92% valid12/12 cases · 0 failed attempts | 83% | 83% | 83% | 67% | 10.52 s | $4.7403per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 92% valid12/12 cases · 0 failed attempts | 92% | 83% | 100% | 67% | 11.91 s | $11.6632per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid12/12 cases · 0 failed attempts | 92% | 100% | 83% | 67% | 21.75 s | $3.0450per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
One pass per case; no confidence intervals or stability claims. Aggregate cost is extrapolated to 1,000 retained cases so incomplete runs do not appear artificially cheap. Rows with fewer than 12 responses still have a smaller case mix and are not directly comparable to complete rows.
Methodology
How UI integrity is scored
Paired clean and defective scenes cover six objective layout-failure types.
Detection
Balanced accuracy, recall, specificity, and false-positive rate.
Classification
Defect-category accuracy after successful detection.
Localization
Predicted box IoU against browser-derived defect regions.
Frozen corpus
Three desktop and three mobile screens
Each fixture has a clean control and one deliberately defective variant. Captures use fixed viewports, local assets, stable target geometry, and exact pixel-difference bounds.
Model dossiers
One evidence page per model
Open a model page for its grounding, element-presence, layout-defect, latency, reliability, and cost results. Routes remain pinned to the named upstream provider with fallbacks disabled.
H Company Holo · FlagshipHolo3 122B A10BFull benchmark results →xAI Grok · Single tierGrok 4.5Full benchmark results →OpenAI GPT · AdditionalGPT-5.4Full benchmark results →Google Gemini · EfficientGemini 3.6 FlashFull benchmark results →OpenAI GPT · FlagshipGPT-5.6 SolFull benchmark results →OpenAI GPT · EfficientGPT-5.6 LunaFull benchmark results →Anthropic Claude · EfficientClaude Sonnet 5Full benchmark results →Anthropic Claude · FlagshipClaude Opus 5Full benchmark results →Google Gemini · FlagshipGemini 3.1 Pro PreviewFull benchmark results →
Methodology
Measure what the model actually sees
The experiment sections document their own scoring rules. This shared methodology covers the common input contract, provider operations, research context, and limits.
InputScreenshot only
Geometry0–1000
RunsOne per case
ScoringSeparate metrics
Input contract
Screenshot only
Prompt text plus one base64 PNG—no DOM, accessibility tree, source, OCR, tools, plugins, or web search.
Fixed rendering
Chromium at 1440 × 900 or 390 × 844; device scale factor 1.
Normalized geometry
Browser-derived ground truth in a 0–1000 coordinate space.
Stable pairs
Clean and defective variants preserve target geometry and capture hashes.
Operations
Keep provider behavior visible
Pinned model IDs, first-party routes, disabled fallbacks, and one response contract.
Retry policy
One retry for 429, 5xx, or network failure—not wrong answers.
Operational metrics
Latency, failures, retries, and cost stay separate from accuracy.
Privacy
Published records omit keys and provider request identifiers.
Limits, data, and license
A pilot, not a leaderboard
Six synthetic screens, one run per case, and 24 targets cannot establish broad superiority or stability. The evidence is published so the scope and individual responses remain inspectable.
Each edition records exact model IDs, date, prompt and schema hashes, endpoint controls, and sanitized attempts because providers change. Raw model records omit API keys, request identifiers, and generation identifiers.
Repeato-created screenshots, annotations, manifests, and aggregate results are available under CC BY 4.0. See the benchmark license notice. Raw model outputs remain subject to their providers' terms.