Fast local evaluation
Repeato’s own local vision algorithms handle frequent interactions, visual matching, and regression checks with fast feedback, predictable behavior, and no per-request model cost.
Google Gemini · Flagship · 2026-08-13
Gemini 3.1 Pro Preview vision benchmark
How Gemini 3.1 Pro Preview performed on screenshot-based UI grounding, element-presence, and visible layout-defect tasks relevant to test automation.
Exact API modelgoogle/gemini-3.1-pro-preview
Requests were pinned to Google AI Studio infrastructure through OpenRouter. Fallbacks were disabled for every retained request.
Grounding hit rate88%
Mean box IoU0.632
Presence accuracy92%
Defect accuracy92%
Overall result
Gemini 3.1 Pro Preview across all three UI tasks
Gemini 3.1 Pro Preview hit 21 of 24 requested UI targets. Its balanced accuracy was 92% for element presence and 92% for layout-defect detection.
This dated pilot retained 24 grounding responses and 24 detection responses. No detection requests failed. Results describe this exact API model and test corpus, not general model quality or a consumer chat product.
Relative performance
Where Gemini 3.1 Pro Preview sits among the tested models
Each scale compares the selected model with the strongest and weakest other model. All three metrics come from the same completed 24-case grounding run; stronger is always to the right.
Box IoUHigher is stronger
0.632- Weakest other
- Claude Sonnet 50.043
- Strongest other
- GPT-5.6 Luna0.771
SpeedLower latency is stronger
11.89 s- Weakest other
- Claude Opus 54.24 s
- Strongest other
- Grok 4.51.67 s
CostLower total cost is stronger
$0.0726- Weakest other
- Claude Opus 5$0.2474
- Strongest other
- GPT-5.6 Luna$0.0057
UI grounding
Can Gemini 3.1 Pro Preview locate actionable elements?
Twenty-four targets span six deterministic desktop and mobile screens. A schema-valid point inside the requested target counts as a hit; box IoU measures localization precision.
Detection results
Presence and visible layout defects
Separate balanced tasks measure whether the model rejects absent elements and distinguishes clean controls from screenshots containing one seeded layout defect.
Can the model tell when an element is not there?
Six present and six absent target queries per model. This track measures classification only.
Balanced accuracy92%
Present recall100%
Absent specificity83%
Coverage12/12
Can the model distinguish clean and broken layouts?
Six clean controls and six screenshots with one seeded defect per model. Transport failures remain operational failures, not accuracy answers.
Balanced accuracy92%
Defect recall100%
Clean specificity83%
Coverage12/12
Why this research matters
Using the right kind of vision for each test
Top vision LLMs can reason about complex visual meaning, but they are still too slow and expensive for every test step. This research helps us identify where their semantic capabilities justify that trade-off.
Semantic reasoning when needed
Top vision LLMs are useful for more complicated questions: understanding meaning, checking dynamic content, describing defects, or evaluating concepts that cannot be captured by a simple visual fingerprint.
An evidence-based hybrid
Benchmarking accuracy, latency, reliability, and cost helps us decide which checks should remain local and which benefit enough from an LLM to justify a slower, more expensive evaluation.
Our product strategy
Check out the whole vision benchmark →Use Repeato’s local computer vision as the fast foundation, then bring in top vision LLMs selectively for complicated or semantic reasoning where they provide meaningful additional value.
Test coverage
Six screens, three task types
Every model is evaluated against the same frozen corpus, prompts, response schemas, and normalized 0–1000 geometry.
Commerce catalog
Desktop web with a clean control and a paired container clipping defect.
Analytics dashboard
Desktop web with a clean control and a paired text overflow defect.
IoT control center
Desktop web with a clean control and a paired element overlap defect.