Fast local evaluation
Repeato’s own local vision algorithms handle frequent interactions, visual matching, and regression checks with fast feedback, predictable behavior, and no per-request model cost.
OpenAI GPT · Flagship · 2026-08-13
GPT-5.6 Sol vision benchmark
How GPT-5.6 Sol performed on screenshot-based UI grounding, element-presence, and visible layout-defect tasks relevant to test automation.
Exact API modelopenai/gpt-5.6-sol
Requests were pinned to OpenAI infrastructure through OpenRouter. Fallbacks were disabled for every retained request.
Grounding hit rate96%
Mean box IoU0.677
Presence accuracy100%
Defect accuracy92%
Overall result
GPT-5.6 Sol across all three UI tasks
GPT-5.6 Sol hit 23 of 24 requested UI targets. Its balanced accuracy was 100% for element presence and 92% for layout-defect detection.
This dated pilot retained 24 grounding responses and 24 detection responses. No detection requests failed. Results describe this exact API model and test corpus, not general model quality or a consumer chat product.
Relative performance
Where GPT-5.6 Sol sits among the tested models
Each scale compares the selected model with the strongest and weakest other model. All three metrics come from the same completed 24-case grounding run; stronger is always to the right.
Box IoUHigher is stronger
0.677- Weakest other
- Claude Sonnet 50.043
- Strongest other
- GPT-5.6 Luna0.771
SpeedLower latency is stronger
3.23 s- Weakest other
- Gemini 3.1 Pro Preview11.89 s
- Strongest other
- Grok 4.51.67 s
CostLower total cost is stronger
$0.2453- Weakest other
- Claude Opus 5$0.2474
- Strongest other
- GPT-5.6 Luna$0.0057
UI grounding
Can GPT-5.6 Sol locate actionable elements?
Twenty-four targets span six deterministic desktop and mobile screens. A schema-valid point inside the requested target counts as a hit; box IoU measures localization precision.
Detection results
Presence and visible layout defects
Separate balanced tasks measure whether the model rejects absent elements and distinguishes clean controls from screenshots containing one seeded layout defect.
Can the model tell when an element is not there?
Six present and six absent target queries per model. This track measures classification only.
Balanced accuracy100%
Present recall100%
Absent specificity100%
Coverage12/12
Can the model distinguish clean and broken layouts?
Six clean controls and six screenshots with one seeded defect per model. Transport failures remain operational failures, not accuracy answers.
Balanced accuracy92%
Defect recall83%
Clean specificity100%
Coverage12/12
Why this research matters
Using the right kind of vision for each test
Top vision LLMs can reason about complex visual meaning, but they are still too slow and expensive for every test step. This research helps us identify where their semantic capabilities justify that trade-off.
Semantic reasoning when needed
Top vision LLMs are useful for more complicated questions: understanding meaning, checking dynamic content, describing defects, or evaluating concepts that cannot be captured by a simple visual fingerprint.
An evidence-based hybrid
Benchmarking accuracy, latency, reliability, and cost helps us decide which checks should remain local and which benefit enough from an LLM to justify a slower, more expensive evaluation.
Our product strategy
Check out the whole vision benchmark →Use Repeato’s local computer vision as the fast foundation, then bring in top vision LLMs selectively for complicated or semantic reasoning where they provide meaningful additional value.
Test coverage
Six screens, three task types
Every model is evaluated against the same frozen corpus, prompts, response schemas, and normalized 0–1000 geometry.
Commerce catalog
Desktop web with a clean control and a paired container clipping defect.
Analytics dashboard
Desktop web with a clean control and a paired text overflow defect.
IoT control center
Desktop web with a clean control and a paired element overlap defect.