Finding controls precisely
GPT-6.1 Sol hit all 72 requested targets across three repetitions, with 81.2% mean target-area overlap. Four configurations hit every target; their predicted areas differed in precision.
Repeato research · LLM Vision benchmark
Can Multimodal LLMs Be Trusted with UI Testing?
A screenshot-only benchmark for UI test automation: locate actionable elements, recognize when an element is absent, detect visible layout defects, and record the operational cost of getting an answer.
2026-10-17
Six deterministic screens. Twenty-four grounding targets. 10 API configurations. Three repetitions per case. A focused pilot, not a ranking.
Screens6
Grounding targets24
Defect categories6
API configurations10
Pilot findings
What this edition shows
Three repetitions per case on six synthetic screens reveal differences worth investigating. They do not establish general model rankings or production reliability.
Presence checks reached the ceiling
All ten configurations answered all 36 presence checks correctly. This small set checks rejection of absent controls, but cannot distinguish the models on accuracy.
Noticing a defect is only the first step
Jev identified all 36 clean or broken screens, but named the exact defect in 12 of 18 defective-screen attempts. Those attempts repeat six distinct defects three times.
What this means for test automation
Use the right kind of vision for each test step
Exact interaction, continuous observation, and semantic interpretation have different performance requirements. One vision technique should not be expected to optimize all three.
Local observationUp to 20 times per secondRemote model requestMedian 0.3–12 seconds in this pilotHow we combine local vision and LLMsCollapse analysis
Repeato's local computer-vision loop can inspect the screen at 20 Hz—once every 50 ms. This pilot measured median remote response times from about 0.3 to 12 seconds across different workloads. The shared presence task allows a fairer timing comparison; a fast classification answer does not imply equally fast target localization. These local and remote figures come from different measurements, not a controlled head-to-head test.
Fast feedback for repeated interaction
Local visual fingerprints can continuously search for recorded targets and act as soon as the screen is ready. Repeating remote model calls at the same frequency would be impractical.
Predictable, private, and inexpensive
Local matching has no network round trip, provider availability, or per-request API charge. Screens remain on the workstation, and the same algorithm can run offline.
AI for questions matching cannot answer
LLM vision earns its latency when a test needs semantic judgment: interpreting text or numbers, understanding dynamic content, or checking concepts such as an upward-trending graph.
Practical split
Use local computer vision for the high-frequency control loop—finding targets, waiting, interaction, and visual comparison. Escalate selectively to AI vision when the assertion genuinely requires language or semantic understanding.
Experiments
Three distinct UI vision tasks
Grounding, element presence, and UI integrity answer different questions and use separate metrics. The pilot does not combine them into an overall score.
Experiment 1 · UI grounding
Can a model locate the requested UI element?
24 targets × three repetitions = 72 attempts per geometry-capable configuration. Invalid and failed responses count as misses.
Nine configurations can return target coordinates. GPT-6 Luna Decisions answers classification questions and is excluded from target-localization results.
- GPT-6 Luna0.796
- GPT-6.1 Sol0.812
- Claude Sonnet 5.50.748
- Gemini 3.8 Flash0.733
- Grok 4.70.477
- Qwen3.8 Max (0902)0.760
- Muse Spark 1.30.630
- GPT-5.6 Luna0.786
- Jev 1.140.581
- GPT-6 Luna$0.0096
- GPT-6.1 Sol$0.1447
- Claude Sonnet 5.5$0.2970
- Gemini 3.8 Flash$0.1007
- Grok 4.7$0.6647
- Qwen3.8 Max (0902)$0.3254
- Muse Spark 1.3$0.2220
- GPT-5.6 Luna$0.0258
- Jev 1.14$0.0079
- GPT-6 Luna2.35 s
- GPT-6.1 Sol2.91 s
- Claude Sonnet 5.52.32 s
- Gemini 3.8 Flash2.75 s
- Grok 4.716.91 s
- Qwen3.8 Max (0902)8.89 s
- Muse Spark 1.34.80 s
- GPT-5.6 Luna2.93 s
- Jev 1.140.37 s
Accuracy and cost trade-offHigher and further left is better.
Methodology
How UI grounding is scored
Four targets per screenshot across common actionable UI-element types.
Point accuracy
Predicted click point falls inside the target box.
Box accuracy
Mean IoU and share of predictions with IoU ≥ 0.5.
Validity
Schema-valid response rate; three repetitions per case reveal within-case consistency.
Experiment 2 · Element presence
Can a model tell when an element is not there?
Six present and six absent targets, each checked three times. This tests visible presence, not completion of a navigation goal.
| 100% valid36/36 cases · 0 failed attempts | 100% | 100% | 100% | 0% | 1.50 s | $0.0752per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid36/36 cases · 0 failed attempts | 100% | 100% | 100% | 0% | 2.24 s | $1.4487per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid36/36 cases · 0 failed attempts | 100% | 100% | 100% | 0% | 2.02 s | $2.9955per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid36/36 cases · 0 failed attempts | 100% | 100% | 100% | 0% | 2.44 s | $1.2008per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid36/36 cases · 0 failed attempts | 100% | 100% | 100% | 0% | 2.66 s | $2.8797per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid36/36 cases · 0 failed attempts | 100% | 100% | 100% | 0% | 4.81 s | $2.6708per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid36/36 cases · 0 failed attempts | 100% | 100% | 100% | 0% | 2.47 s | $2.0156per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid36/36 cases · 0 failed attempts | 100% | 100% | 100% | 0% | 1.59 s | $0.1533per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid36/36 cases · 0 failed attempts | 100% | 100% | 100% | 0% | 0.38 s | $0.1069per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid36/36 cases · 0 failed attempts | 100% | 100% | 100% | 0% | 0.39 s | $0.1062per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Three repetitions per case on six synthetic screens. Costs include reported cache discounts. These results do not establish production reliability or general model rankings.
Methodology
How element presence is scored
One known-present and one plausible-but-absent query per clean screen.
Detection
Balanced accuracy weights present and absent cases equally.
Class behavior
Recall, specificity, and false-positive rate expose error direction.
Boolean only
found=true or found=false; no localization score.
Experiment 3 · UI integrity
Can a model distinguish clean and broken layouts?
Six clean and six seeded defective screens × three repetitions. Recognizing a broken screen, naming its exact defect, and locating its region are separate results.
Defect accuracy and cost trade-offHigher and further left is better.
Defect accuracy and latency trade-offHigher and further left is better. Latency is the mean response time per retained defect case. Classification-only Decisions does not produce regions.
| 100% valid36/36 cases · 0 failed attempts | 92% | 83% | 100% | 56% | 3.04 s | $0.1751per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid36/36 cases · 0 failed attempts | 92% | 83% | 100% | 33% | 3.64 s | $2.3176per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 83% valid36/36 cases · 0 failed attempts | 83% | 67% | 100% | 50% | 2.87 s | $4.8735per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 97% valid36/36 cases · 0 failed attempts | 78% | 100% | 56% | 72% | 7.06 s | $3.3765per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 97% valid36/36 cases · 0 failed attempts | 94% | 94% | 94% | 72% | 17.48 s | $9.6872per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 97% valid36/36 cases · 0 failed attempts | 94% | 94% | 94% | 39% | 11.78 s | $5.3957per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid36/36 cases · 0 failed attempts | 89% | 78% | 100% | 56% | 6.34 s | $3.8791per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid36/36 cases · 0 failed attempts | 92% | 83% | 100% | 67% | 2.88 s | $0.3518per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid36/36 cases · 0 failed attempts | 100% | 100% | 100% | 67% | 0.37 s | $0.1166per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100% valid36/36 cases · 0 failed attempts | 83% | 67% | 100% | 50% | 0.42 s | $0.1291per 1K retained cases | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Three repetitions per case on six synthetic screens. Costs include reported cache discounts. These results do not establish production reliability or general model rankings.
The existing defect scorer assigns zero region overlap when the category is wrong. Luna Decisions is evaluated with a classification-only contract; it is not required to produce a region.
Methodology
How UI integrity is scored
Paired clean and defective scenes cover six objective layout-failure types.
Detection
Balanced accuracy, recall, specificity, and false-positive rate.
Classification
Defect-category accuracy after successful detection.
Localization
Predicted box IoU against browser-derived defect regions.
Frozen corpus
Three desktop and three mobile screens
Each fixture has a clean control and one deliberately defective variant. Captures use fixed viewports, local assets, stable target geometry, and exact pixel-difference bounds.
Model dossiers
Inspect every retained case
Each repetition remains visible, including incorrect, invalid, and failed attempts. Expand a model to inspect its cases; the raw download retains predictions and native decisions.
openaiGPT-6 LunaFull benchmark results →openaiGPT-6.1 SolFull benchmark results →anthropicClaude Sonnet 5.5Full benchmark results →googleGemini 3.8 FlashFull benchmark results →x-aiGrok 4.7Full benchmark results →qwenQwen3.8 Max (0902)Full benchmark results →metaMuse Spark 1.3Full benchmark results →openaiGPT-5.6 LunaFull benchmark results →typesafeJev 1.14Full benchmark results →openaiGPT-6 Luna DecisionsFull benchmark results →
GPT-6 Luna · 144 case executions
Returned model versions: openai/gpt-6-luna.
Targets with the same hit/miss outcome across all three repetitions: 23 / 24. Agreement can include consistently missing a target.
| Case | Repeat | Result | Outcome | Time |
|---|---|---|---|---|
| commerce-catalog-desktop/clean/grounding/search | 1 | success | Target hit | 3.33 s |
| commerce-catalog-desktop/clean/grounding/add-product | 1 | success | Target hit | 1.92 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 1 | success | Target hit | 2.65 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 1 | success | Target hit | 3.77 s |
| commerce-catalog-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.33 s |
| commerce-catalog-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 191, "y": 145, "width": 69, "height": 46 }, "description": "The page heading is clipped on the left, showing only “ducts” instead of the full title." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 191, "y": 145, "width": 69, "height": 46 }, "description": "The page heading is clipped on the left, showing only “ducts” instead of the full title." } | 3.98 s |
| commerce-catalog-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.55 s |
| commerce-catalog-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.28 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 1 | success | Target hit | 2.66 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 1 | success | Target hit | 2.42 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 1 | success | Target hit | 2.01 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 1 | success | Target hit | 2.24 s |
| analytics-dashboard-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.99 s |
| analytics-dashboard-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 248, "width": 326, "height": 31 }, "description": "The revenue card heading is clipped at the card’s right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 248, "width": 326, "height": 31 }, "description": "The revenue card heading is clipped at the card’s right edge." } | 2.59 s |
| analytics-dashboard-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.57 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.57 s |
| iot-control-desktop/clean/grounding/devices-nav | 1 | success | Target hit | 2.21 s |
| iot-control-desktop/clean/grounding/restart-device | 1 | success | Target hit | 1.98 s |
| iot-control-desktop/clean/grounding/auto-mode | 1 | success | Target hit | 2.62 s |
| iot-control-desktop/clean/grounding/pump-card | 1 | success | Target hit | 2.80 s |
| iot-control-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.04 s |
| iot-control-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 222, "height": 64 }, "description": "The maintenance notification overlaps the device table rows." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 222, "height": 64 }, "description": "The maintenance notification overlaps the device table rows." } | 3.62 s |
| iot-control-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.49 s |
| iot-control-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.57 s |
| banking-ios/clean/grounding/profile | 1 | success | Target hit | 2.46 s |
| banking-ios/clean/grounding/transaction-search | 1 | success | Target hit | 2.81 s |
| banking-ios/clean/grounding/hide-balance | 1 | success | Target hit | 2.59 s |
| banking-ios/clean/grounding/transfer | 1 | success | Target hit | 1.87 s |
| banking-ios/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.16 s |
| banking-ios/defect/layout-defect | 1 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 6.07 s |
| banking-ios/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.72 s |
| banking-ios/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.24 s |
| flight-pass-ios/clean/grounding/boarding-pass | 1 | success | Target hit | 2.49 s |
| flight-pass-ios/clean/grounding/check-in | 1 | success | Target hit | 2.78 s |
| flight-pass-ios/clean/grounding/trips-tab | 1 | success | Target hit | 1.63 s |
| flight-pass-ios/clean/grounding/profile-tab | 1 | success | Target hit | 2.53 s |
| flight-pass-ios/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.76 s |
| flight-pass-ios/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "A red flight-update banner covers the lower portion of the blue action button." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "A red flight-update banner covers the lower portion of the blue action button." } | 2.79 s |
| flight-pass-ios/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.24 s |
| flight-pass-ios/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.12 s |
| smart-home-android/clean/grounding/more | 1 | success | Target hit | 1.47 s |
| smart-home-android/clean/grounding/room-search | 1 | success | Target hit | 1.88 s |
| smart-home-android/clean/grounding/thermostat-card | 1 | success | Target hit | 1.46 s |
| smart-home-android/clean/grounding/add-device | 1 | success | Target hit | 1.38 s |
| smart-home-android/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.93 s |
| smart-home-android/defect/layout-defect | 1 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 4.04 s |
| smart-home-android/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.19 s |
| smart-home-android/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.21 s |
| commerce-catalog-desktop/clean/grounding/search | 2 | success | Target hit | 2.62 s |
| commerce-catalog-desktop/clean/grounding/add-product | 2 | success | Target hit | 2.12 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 2 | success | Target hit | 1.54 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 2 | success | Target hit | 2.71 s |
| commerce-catalog-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.88 s |
| commerce-catalog-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 151, "width": 68, "height": 44 }, "description": "The page heading is clipped on the left; only “ducts” is visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 151, "width": 68, "height": 44 }, "description": "The page heading is clipped on the left; only “ducts” is visible." } | 3.44 s |
| commerce-catalog-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.75 s |
| commerce-catalog-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.65 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 2 | success | Target hit | 3.18 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 2 | success | Target hit | 2.89 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 2 | success | Target hit | 2.21 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 2 | success | Target hit | 2.67 s |
| analytics-dashboard-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.91 s |
| analytics-dashboard-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 253, "width": 326, "height": 37 }, "description": "The revenue card heading is cut off at the right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 253, "width": 326, "height": 37 }, "description": "The revenue card heading is cut off at the right edge." } | 2.65 s |
| analytics-dashboard-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.97 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.67 s |
| iot-control-desktop/clean/grounding/devices-nav | 2 | success | Target hit | 2.20 s |
| iot-control-desktop/clean/grounding/restart-device | 2 | success | Target hit | 2.25 s |
| iot-control-desktop/clean/grounding/auto-mode | 2 | success | Target hit | 2.86 s |
| iot-control-desktop/clean/grounding/pump-card | 2 | success | Target hit | 3.25 s |
| iot-control-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.92 s |
| iot-control-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 222, "height": 64 }, "description": "The maintenance notification overlaps the device table rows." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 222, "height": 64 }, "description": "The maintenance notification overlaps the device table rows." } | 3.14 s |
| iot-control-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.77 s |
| iot-control-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.64 s |
| banking-ios/clean/grounding/profile | 2 | success | Target hit | 3.40 s |
| banking-ios/clean/grounding/transaction-search | 2 | success | Target hit | 2.17 s |
| banking-ios/clean/grounding/hide-balance | 2 | success | Target hit | 2.38 s |
| banking-ios/clean/grounding/transfer | 2 | success | Target hit | 1.75 s |
| banking-ios/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.87 s |
| banking-ios/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 72, "width": 240, "height": 32 }, "description": "The first letter of the Banking heading is clipped at the left edge of its container." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 72, "width": 240, "height": 32 }, "description": "The first letter of the Banking heading is clipped at the left edge of its container." } | 5.97 s |
| banking-ios/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.17 s |
| banking-ios/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.45 s |
| flight-pass-ios/clean/grounding/boarding-pass | 2 | success | Target hit | 2.26 s |
| flight-pass-ios/clean/grounding/check-in | 2 | success | Target hit | 1.47 s |
| flight-pass-ios/clean/grounding/trips-tab | 2 | success | Target hit | 1.62 s |
| flight-pass-ios/clean/grounding/profile-tab | 2 | success | Target missed | 1.55 s |
| flight-pass-ios/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.20 s |
| flight-pass-ios/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner obscures part of the blue actionable button beneath it." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner obscures part of the blue actionable button beneath it." } | 3.68 s |
| flight-pass-ios/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.22 s |
| flight-pass-ios/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.09 s |
| smart-home-android/clean/grounding/more | 2 | success | Target hit | 1.72 s |
| smart-home-android/clean/grounding/room-search | 2 | success | Target hit | 2.22 s |
| smart-home-android/clean/grounding/thermostat-card | 2 | success | Target hit | 1.53 s |
| smart-home-android/clean/grounding/add-device | 2 | success | Target hit | 1.48 s |
| smart-home-android/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.82 s |
| smart-home-android/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 354, "y": 510, "width": 646, "height": 69 }, "description": "The second Quick view card extends beyond the right edge of the viewport and is cropped." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 354, "y": 510, "width": 646, "height": 69 }, "description": "The second Quick view card extends beyond the right edge of the viewport and is cropped." } | 4.63 s |
| smart-home-android/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.18 s |
| smart-home-android/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.45 s |
| commerce-catalog-desktop/clean/grounding/search | 3 | success | Target hit | 2.81 s |
| commerce-catalog-desktop/clean/grounding/add-product | 3 | success | Target hit | 1.70 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 3 | success | Target hit | 2.45 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 3 | success | Target hit | 3.83 s |
| commerce-catalog-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 3.66 s |
| commerce-catalog-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 191, "y": 150, "width": 70, "height": 40 }, "description": "The page title is visibly clipped at its beginning, showing only “ducts”." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 191, "y": 150, "width": 70, "height": 40 }, "description": "The page title is visibly clipped at its beginning, showing only “ducts”." } | 3.67 s |
| commerce-catalog-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.85 s |
| commerce-catalog-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.33 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 3 | success | Target hit | 3.45 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 3 | success | Target hit | 2.06 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 3 | success | Target hit | 2.00 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 3 | success | Target hit | 2.30 s |
| analytics-dashboard-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.75 s |
| analytics-dashboard-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 253, "width": 326, "height": 32 }, "description": "The monthly revenue card heading is clipped at the card’s right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 253, "width": 326, "height": 32 }, "description": "The monthly revenue card heading is clipped at the card’s right edge." } | 3.12 s |
| analytics-dashboard-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.27 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.32 s |
| iot-control-desktop/clean/grounding/devices-nav | 3 | success | Target hit | 2.50 s |
| iot-control-desktop/clean/grounding/restart-device | 3 | success | Target hit | 2.75 s |
| iot-control-desktop/clean/grounding/auto-mode | 3 | success | Target hit | 2.79 s |
| iot-control-desktop/clean/grounding/pump-card | 3 | success | Target hit | 2.32 s |
| iot-control-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 3.03 s |
| iot-control-desktop/defect/layout-defect | 3 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 3.08 s |
| iot-control-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.44 s |
| iot-control-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.71 s |
| banking-ios/clean/grounding/profile | 3 | success | Target hit | 2.41 s |
| banking-ios/clean/grounding/transaction-search | 3 | success | Target hit | 2.90 s |
| banking-ios/clean/grounding/hide-balance | 3 | success | Target hit | 2.29 s |
| banking-ios/clean/grounding/transfer | 3 | success | Target hit | 2.51 s |
| banking-ios/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.21 s |
| banking-ios/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 0, "y": 69, "width": 285, "height": 35 }, "description": "The heading at the upper left is clipped on its left side." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 0, "y": 69, "width": 285, "height": 35 }, "description": "The heading at the upper left is clipped on its left side." } | 5.43 s |
| banking-ios/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.45 s |
| banking-ios/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.26 s |
| flight-pass-ios/clean/grounding/boarding-pass | 3 | success | Target hit | 2.16 s |
| flight-pass-ios/clean/grounding/check-in | 3 | success | Target hit | 2.23 s |
| flight-pass-ios/clean/grounding/trips-tab | 3 | success | Target hit | 2.66 s |
| flight-pass-ios/clean/grounding/profile-tab | 3 | success | Target hit | 3.25 s |
| flight-pass-ios/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.77 s |
| flight-pass-ios/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner obscures part of the blue actionable button beneath it." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner obscures part of the blue actionable button beneath it." } | 2.89 s |
| flight-pass-ios/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.24 s |
| flight-pass-ios/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.06 s |
| smart-home-android/clean/grounding/more | 3 | success | Target hit | 2.34 s |
| smart-home-android/clean/grounding/room-search | 3 | success | Target hit | 2.18 s |
| smart-home-android/clean/grounding/thermostat-card | 3 | success | Target hit | 1.72 s |
| smart-home-android/clean/grounding/add-device | 3 | success | Target hit | 1.51 s |
| smart-home-android/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.43 s |
| smart-home-android/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 354, "y": 510, "width": 646, "height": 69 }, "description": "The Kitchen lights card is cut off at the right edge of the viewport." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 354, "y": 510, "width": 646, "height": 69 }, "description": "The Kitchen lights card is cut off at the right edge of the viewport." } | 5.87 s |
| smart-home-android/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.47 s |
| smart-home-android/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.66 s |
GPT-6.1 Sol · 144 case executions
Returned model versions: openai/gpt-6.1-sol.
Targets with the same hit/miss outcome across all three repetitions: 24 / 24. Agreement can include consistently missing a target.
| Case | Repeat | Result | Outcome | Time |
|---|---|---|---|---|
| commerce-catalog-desktop/clean/grounding/search | 1 | success | Target hit | 3.73 s |
| commerce-catalog-desktop/clean/grounding/add-product | 1 | success | Target hit | 3.41 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 1 | success | Target hit | 2.75 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 1 | success | Target hit | 3.00 s |
| commerce-catalog-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 3.79 s |
| commerce-catalog-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 68, "height": 68 }, "description": "The page heading and its small label are clipped on the left, leaving only trailing portions of the text visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 68, "height": 68 }, "description": "The page heading and its small label are clipped on the left, leaving only trailing portions of the text visible." } | 5.65 s |
| commerce-catalog-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.04 s |
| commerce-catalog-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.52 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 1 | success | Target hit | 4.02 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 1 | success | Target hit | 3.04 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 1 | success | Target hit | 3.42 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 1 | success | Target hit | 3.78 s |
| analytics-dashboard-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.68 s |
| analytics-dashboard-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 250, "width": 326, "height": 31 }, "description": "The monthly revenue card heading is clipped at the card's right edge." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 250, "width": 326, "height": 31 }, "description": "The monthly revenue card heading is clipped at the card's right edge." } | 3.75 s |
| analytics-dashboard-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.18 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.13 s |
| iot-control-desktop/clean/grounding/devices-nav | 1 | success | Target hit | 3.30 s |
| iot-control-desktop/clean/grounding/restart-device | 1 | success | Target hit | 2.72 s |
| iot-control-desktop/clean/grounding/auto-mode | 1 | success | Target hit | 2.67 s |
| iot-control-desktop/clean/grounding/pump-card | 1 | success | Target hit | 2.61 s |
| iot-control-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.52 s |
| iot-control-desktop/defect/layout-defect | 1 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.67 s |
| iot-control-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.46 s |
| iot-control-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.27 s |
| banking-ios/clean/grounding/profile | 1 | success | Target hit | 2.88 s |
| banking-ios/clean/grounding/transaction-search | 1 | success | Target hit | 2.62 s |
| banking-ios/clean/grounding/hide-balance | 1 | success | Target hit | 2.78 s |
| banking-ios/clean/grounding/transfer | 1 | success | Target hit | 3.09 s |
| banking-ios/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.96 s |
| banking-ios/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 47, "width": 238, "height": 59 }, "description": "The brand label and Banking heading are clipped along their left edge, cutting off their initial letters." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 47, "width": 238, "height": 59 }, "description": "The brand label and Banking heading are clipped along their left edge, cutting off their initial letters." } | 8.64 s |
| banking-ios/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.87 s |
| banking-ios/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.35 s |
| flight-pass-ios/clean/grounding/boarding-pass | 1 | success | Target hit | 2.48 s |
| flight-pass-ios/clean/grounding/check-in | 1 | success | Target hit | 4.77 s |
| flight-pass-ios/clean/grounding/trips-tab | 1 | success | Target hit | 2.43 s |
| flight-pass-ios/clean/grounding/profile-tab | 1 | success | Target hit | 2.85 s |
| flight-pass-ios/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.39 s |
| flight-pass-ios/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 64 }, "description": "The flight update banner overlaps the bottom blue action button, obscuring its label and lower portion." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 64 }, "description": "The flight update banner overlaps the bottom blue action button, obscuring its label and lower portion." } | 3.86 s |
| flight-pass-ios/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.98 s |
| flight-pass-ios/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.53 s |
| smart-home-android/clean/grounding/more | 1 | success | Target hit | 2.27 s |
| smart-home-android/clean/grounding/room-search | 1 | success | Target hit | 2.44 s |
| smart-home-android/clean/grounding/thermostat-card | 1 | success | Target hit | 2.78 s |
| smart-home-android/clean/grounding/add-device | 1 | success | Target hit | 2.72 s |
| smart-home-android/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.44 s |
| smart-home-android/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 70 }, "description": "The Kitchen lights card is shifted right and clipped at the list's right edge, cutting off its right side and trailing control." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 70 }, "description": "The Kitchen lights card is shifted right and clipped at the list's right edge, cutting off its right side and trailing control." } | 5.47 s |
| smart-home-android/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.67 s |
| smart-home-android/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.36 s |
| commerce-catalog-desktop/clean/grounding/search | 2 | success | Target hit | 2.80 s |
| commerce-catalog-desktop/clean/grounding/add-product | 2 | success | Target hit | 2.61 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 2 | success | Target hit | 2.82 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 2 | success | Target hit | 3.15 s |
| commerce-catalog-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 3.00 s |
| commerce-catalog-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 73, "height": 68 }, "description": "The page heading and its section label are clipped on the left, hiding their initial characters." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 73, "height": 68 }, "description": "The page heading and its section label are clipped on the left, hiding their initial characters." } | 4.90 s |
| commerce-catalog-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.41 s |
| commerce-catalog-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.60 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 2 | success | Target hit | 2.80 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 2 | success | Target hit | 2.48 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 2 | success | Target hit | 2.85 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 2 | success | Target hit | 3.92 s |
| analytics-dashboard-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.98 s |
| analytics-dashboard-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 249, "width": 326, "height": 32 }, "description": "The monthly revenue heading is cut off at the right edge of its card." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 249, "width": 326, "height": 32 }, "description": "The monthly revenue heading is cut off at the right edge of its card." } | 3.47 s |
| analytics-dashboard-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.60 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.42 s |
| iot-control-desktop/clean/grounding/devices-nav | 2 | success | Target hit | 3.34 s |
| iot-control-desktop/clean/grounding/restart-device | 2 | success | Target hit | 2.94 s |
| iot-control-desktop/clean/grounding/auto-mode | 2 | success | Target hit | 4.14 s |
| iot-control-desktop/clean/grounding/pump-card | 2 | success | Target hit | 2.51 s |
| iot-control-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.47 s |
| iot-control-desktop/defect/layout-defect | 2 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.90 s |
| iot-control-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.94 s |
| iot-control-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.50 s |
| banking-ios/clean/grounding/profile | 2 | success | Target hit | 2.89 s |
| banking-ios/clean/grounding/transaction-search | 2 | success | Target hit | 2.58 s |
| banking-ios/clean/grounding/hide-balance | 2 | success | Target hit | 2.40 s |
| banking-ios/clean/grounding/transfer | 2 | success | Target hit | 2.81 s |
| banking-ios/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.89 s |
| banking-ios/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 69, "width": 241, "height": 38 }, "description": "The left edge of the Banking heading is clipped, cutting off part of its first letter." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 69, "width": 241, "height": 38 }, "description": "The left edge of the Banking heading is clipped, cutting off part of its first letter." } | 7.42 s |
| banking-ios/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.72 s |
| banking-ios/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.02 s |
| flight-pass-ios/clean/grounding/boarding-pass | 2 | success | Target hit | 3.14 s |
| flight-pass-ios/clean/grounding/check-in | 2 | success | Target hit | 2.86 s |
| flight-pass-ios/clean/grounding/trips-tab | 2 | success | Target hit | 2.24 s |
| flight-pass-ios/clean/grounding/profile-tab | 2 | success | Target hit | 2.47 s |
| flight-pass-ios/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.99 s |
| flight-pass-ios/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 64 }, "description": "The red flight-update banner overlaps the blue bottom action button, obscuring its label and much of its clickable area." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 64 }, "description": "The red flight-update banner overlaps the blue bottom action button, obscuring its label and much of its clickable area." } | 4.43 s |
| flight-pass-ios/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.94 s |
| flight-pass-ios/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.89 s |
| smart-home-android/clean/grounding/more | 2 | success | Target hit | 2.35 s |
| smart-home-android/clean/grounding/room-search | 2 | success | Target hit | 2.58 s |
| smart-home-android/clean/grounding/thermostat-card | 2 | success | Target hit | 2.82 s |
| smart-home-android/clean/grounding/add-device | 2 | success | Target hit | 2.71 s |
| smart-home-android/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.13 s |
| smart-home-android/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 71 }, "description": "The Kitchen lights row is shifted right and clipped at the content boundary, cutting off its right edge and trailing control." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 71 }, "description": "The Kitchen lights row is shifted right and clipped at the content boundary, cutting off its right edge and trailing control." } | 5.02 s |
| smart-home-android/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.05 s |
| smart-home-android/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.98 s |
| commerce-catalog-desktop/clean/grounding/search | 3 | success | Target hit | 3.93 s |
| commerce-catalog-desktop/clean/grounding/add-product | 3 | success | Target hit | 2.40 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 3 | success | Target hit | 3.39 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 3 | success | Target hit | 2.85 s |
| commerce-catalog-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the specified categories." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the specified categories." } | 3.40 s |
| commerce-catalog-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 70, "height": 68 }, "description": "The page heading and its overline are clipped on the left, leaving only trailing portions of the text visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 70, "height": 68 }, "description": "The page heading and its overline are clipped on the left, leaving only trailing portions of the text visible." } | 4.58 s |
| commerce-catalog-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.23 s |
| commerce-catalog-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 3.35 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 3 | success | Target hit | 2.94 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 3 | success | Target hit | 2.86 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 3 | success | Target hit | 2.81 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 3 | success | Target hit | 3.48 s |
| analytics-dashboard-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.59 s |
| analytics-dashboard-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 249, "width": 327, "height": 32 }, "description": "The monthly revenue heading is cut off at the right edge of its card." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 249, "width": 327, "height": 32 }, "description": "The monthly revenue heading is cut off at the right edge of its card." } | 3.59 s |
| analytics-dashboard-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.55 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.03 s |
| iot-control-desktop/clean/grounding/devices-nav | 3 | success | Target hit | 3.04 s |
| iot-control-desktop/clean/grounding/restart-device | 3 | success | Target hit | 3.26 s |
| iot-control-desktop/clean/grounding/auto-mode | 3 | success | Target hit | 2.99 s |
| iot-control-desktop/clean/grounding/pump-card | 3 | success | Target hit | 2.80 s |
| iot-control-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 3.12 s |
| iot-control-desktop/defect/layout-defect | 3 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.84 s |
| iot-control-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.39 s |
| iot-control-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.19 s |
| banking-ios/clean/grounding/profile | 3 | success | Target hit | 3.35 s |
| banking-ios/clean/grounding/transaction-search | 3 | success | Target hit | 2.37 s |
| banking-ios/clean/grounding/hide-balance | 3 | success | Target hit | 3.37 s |
| banking-ios/clean/grounding/transfer | 3 | success | Target hit | 2.54 s |
| banking-ios/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.05 s |
| banking-ios/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 44, "y": 68, "width": 245, "height": 39 }, "description": "The left edge of the Banking heading is clipped, cutting off part of its first letter." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 44, "y": 68, "width": 245, "height": 39 }, "description": "The left edge of the Banking heading is clipped, cutting off part of its first letter." } | 6.93 s |
| banking-ios/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.85 s |
| banking-ios/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.60 s |
| flight-pass-ios/clean/grounding/boarding-pass | 3 | success | Target hit | 2.12 s |
| flight-pass-ios/clean/grounding/check-in | 3 | success | Target hit | 2.21 s |
| flight-pass-ios/clean/grounding/trips-tab | 3 | success | Target hit | 3.53 s |
| flight-pass-ios/clean/grounding/profile-tab | 3 | success | Target hit | 2.35 s |
| flight-pass-ios/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.03 s |
| flight-pass-ios/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner covers the lower portion and label of the blue action button above the bottom navigation." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner covers the lower portion and label of the blue action button above the bottom navigation." } | 4.19 s |
| flight-pass-ios/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.91 s |
| flight-pass-ios/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.12 s |
| smart-home-android/clean/grounding/more | 3 | success | Target hit | 2.30 s |
| smart-home-android/clean/grounding/room-search | 3 | success | Target hit | 2.60 s |
| smart-home-android/clean/grounding/thermostat-card | 3 | success | Target hit | 2.75 s |
| smart-home-android/clean/grounding/add-device | 3 | success | Target hit | 2.49 s |
| smart-home-android/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.17 s |
| smart-home-android/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 70 }, "description": "The Kitchen lights card is shifted right and clipped at the list container’s right edge, cutting off its right side and trailing control." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 70 }, "description": "The Kitchen lights card is shifted right and clipped at the list container’s right edge, cutting off its right side and trailing control." } | 5.17 s |
| smart-home-android/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.06 s |
| smart-home-android/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.07 s |
Claude Sonnet 5.5 · 144 case executions
Returned model versions: anthropic/claude-sonnet-5.5.
Targets with the same hit/miss outcome across all three repetitions: 22 / 24. Agreement can include consistently missing a target.
| Case | Repeat | Result | Outcome | Time |
|---|---|---|---|---|
| commerce-catalog-desktop/clean/grounding/search | 1 | success | Target hit | 3.96 s |
| commerce-catalog-desktop/clean/grounding/add-product | 1 | invalid response | Target missed | 2.02 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 1 | success | Target hit | 3.08 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 1 | success | Target hit | 2.40 s |
| commerce-catalog-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact and aligned." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact and aligned." } | 3.75 s |
| commerce-catalog-desktop/defect/layout-defect | 1 | invalid response | The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 191, "y": 123, "width": 200, "height": 75 }, "description": "The page heading and eyebrow text are cut off on the left edge. The eyebrow reads 'IANAGEMENT' instead of 'MANAGEMENT', and the title reads 'ducts' instead of 'Products', so the leading characters are clipped by the container." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 191, "y": 123, "width": 200, "height": 75 }, "description": "The page heading and eyebrow text are cut off on the left edge. The eyebrow reads 'IANAGEMENT' instead of 'MANAGEMENT', and the title reads 'ducts' instead of 'Products', so the leading characters are clipped by the container." } | 4.65 s |
| commerce-catalog-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.46 s |
| commerce-catalog-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.24 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 1 | success | Target hit | 2.47 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 1 | success | Target hit | 2.46 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 1 | success | Target hit | 2.94 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 1 | success | Target hit | 2.31 s |
| analytics-dashboard-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible clipping, overflow, overlap, obscured controls, or reflow problems; all text and controls are fully visible and properly contained." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible clipping, overflow, overlap, obscured controls, or reflow problems; all text and controls are fully visible and properly contained." } | 4.48 s |
| analytics-dashboard-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 250, "width": 356, "height": 32 }, "description": "The red card title 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is clipped by the container." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 250, "width": 356, "height": 32 }, "description": "The red card title 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is clipped by the container." } | 2.55 s |
| analytics-dashboard-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.14 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.14 s |
| iot-control-desktop/clean/grounding/devices-nav | 1 | success | Target hit | 3.05 s |
| iot-control-desktop/clean/grounding/restart-device | 1 | success | Target hit | 2.13 s |
| iot-control-desktop/clean/grounding/auto-mode | 1 | success | Target hit | 2.39 s |
| iot-control-desktop/clean/grounding/pump-card | 1 | success | Target hit | 2.31 s |
| iot-control-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact." } | 3.98 s |
| iot-control-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' banner floats over the table rows, covering part of the Cooling pump and Intake valve row areas and overlapping the row divider." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' banner floats over the table rows, covering part of the Cooling pump and Intake valve row areas and overlapping the row divider." } | 4.73 s |
| iot-control-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.99 s |
| iot-control-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.35 s |
| banking-ios/clean/grounding/profile | 1 | success | Target hit | 1.84 s |
| banking-ios/clean/grounding/transaction-search | 1 | success | Target hit | 3.14 s |
| banking-ios/clean/grounding/hide-balance | 1 | success | Target hit | 1.92 s |
| banking-ios/clean/grounding/transfer | 1 | success | Target hit | 1.82 s |
| banking-ios/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all text and controls are fully visible, contained, and not overlapping." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all text and controls are fully visible, contained, and not overlapping." } | 2.21 s |
| banking-ios/defect/layout-defect | 1 | invalid response | The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 300, "height": 115 }, "description": "The header title is cut off at the left screen edge: 'banking' renders as 'oanking' and the 'MAYA' label above it as 'IAYA', so the leading characters are cropped." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 300, "height": 115 }, "description": "The header title is cut off at the left screen edge: 'banking' renders as 'oanking' and the 'MAYA' label above it as 'IAYA', so the leading characters are cropped." } | 3.40 s |
| banking-ios/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.51 s |
| banking-ios/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.83 s |
| flight-pass-ios/clean/grounding/boarding-pass | 1 | success | Target hit | 2.46 s |
| flight-pass-ios/clean/grounding/check-in | 1 | success | Target hit | 1.91 s |
| flight-pass-ios/clean/grounding/trips-tab | 1 | success | Target hit | 1.89 s |
| flight-pass-ios/clean/grounding/profile-tab | 1 | success | Target hit | 2.30 s |
| flight-pass-ios/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; content is fully visible, text fits its containers, and no elements overlap or obscure controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; content is fully visible, text fits its containers, and no elements overlap or obscure controls." } | 2.34 s |
| flight-pass-ios/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 80 }, "description": "A red 'Flight update' banner covers the blue primary button near the bottom, hiding most of its label so only the tops of the letters remain visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 80 }, "description": "A red 'Flight update' banner covers the blue primary button near the bottom, hiding most of its label so only the tops of the letters remain visible." } | 3.21 s |
| flight-pass-ios/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.77 s |
| flight-pass-ios/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 4.33 s |
| smart-home-android/clean/grounding/more | 1 | success | Target hit | 2.23 s |
| smart-home-android/clean/grounding/room-search | 1 | success | Target hit | 9.81 s |
| smart-home-android/clean/grounding/thermostat-card | 1 | success | Target hit | 2.25 s |
| smart-home-android/clean/grounding/add-device | 1 | success | Target hit | 1.83 s |
| smart-home-android/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; all elements are fully rendered and aligned." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; all elements are fully rendered and aligned." } | 4.38 s |
| smart-home-android/defect/layout-defect | 1 | invalid response | The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 533, "width": 646, "height": 46 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its right side is cut off." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 533, "width": 646, "height": 46 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its right side is cut off." } | 5.42 s |
| smart-home-android/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.53 s |
| smart-home-android/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.99 s |
| commerce-catalog-desktop/clean/grounding/search | 2 | success | Target hit | 2.13 s |
| commerce-catalog-desktop/clean/grounding/add-product | 2 | success | Target hit | 1.89 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 2 | success | Target hit | 2.72 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 2 | success | Target hit | 2.05 s |
| commerce-catalog-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." } | 2.41 s |
| commerce-catalog-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 125, "width": 205, "height": 57 }, "description": "The page heading and eyebrow text are cut off at the left edge of their container, showing 'IANAGEMENT' and 'ducts' instead of the full words." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 125, "width": 205, "height": 57 }, "description": "The page heading and eyebrow text are cut off at the left edge of their container, showing 'IANAGEMENT' and 'ducts' instead of the full words." } | 3.99 s |
| commerce-catalog-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.64 s |
| commerce-catalog-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.47 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 2 | success | Target hit | 2.29 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 2 | success | Target hit | 2.15 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 2 | success | Target hit | 2.47 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 2 | success | Target hit | 2.02 s |
| analytics-dashboard-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible. The layout appears intact." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible. The layout appears intact." } | 3.55 s |
| analytics-dashboard-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 249, "width": 355, "height": 40 }, "description": "The red heading 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is clipped by the container." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 249, "width": 355, "height": 40 }, "description": "The red heading 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is clipped by the container." } | 2.22 s |
| analytics-dashboard-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.77 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.68 s |
| iot-control-desktop/clean/grounding/devices-nav | 2 | success | Target hit | 2.64 s |
| iot-control-desktop/clean/grounding/restart-device | 2 | success | Target hit | 2.20 s |
| iot-control-desktop/clean/grounding/auto-mode | 2 | success | Target hit | 2.46 s |
| iot-control-desktop/clean/grounding/pump-card | 2 | success | Target hit | 2.21 s |
| iot-control-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." } | 2.09 s |
| iot-control-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "A red 'Maintenance scheduled' banner floats over the device table, covering parts of the Cooling pump and Intake valve rows and overlapping the row divider." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "A red 'Maintenance scheduled' banner floats over the device table, covering parts of the Cooling pump and Intake valve rows and overlapping the row divider." } | 3.48 s |
| iot-control-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.35 s |
| iot-control-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.89 s |
| banking-ios/clean/grounding/profile | 2 | success | Target hit | 2.23 s |
| banking-ios/clean/grounding/transaction-search | 2 | success | Target hit | 1.61 s |
| banking-ios/clean/grounding/hide-balance | 2 | success | Target hit | 1.87 s |
| banking-ios/clean/grounding/transfer | 2 | success | Target hit | 1.83 s |
| banking-ios/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all text, cards, and controls render fully within their containers without overlap or cropping." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all text, cards, and controls render fully within their containers without overlap or cropping." } | 2.29 s |
| banking-ios/defect/layout-defect | 2 | invalid response | The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 300, "height": 55 }, "description": "The header text at the top left is cut off by the left edge of the viewport: the brand label reads 'IAYA' and the title reads 'anking' instead of 'Banking', so the first letters are cropped." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 300, "height": 55 }, "description": "The header text at the top left is cut off by the left edge of the viewport: the brand label reads 'IAYA' and the title reads 'anking' instead of 'Banking', so the first letters are cropped." } | 2.89 s |
| banking-ios/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.61 s |
| banking-ios/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.63 s |
| flight-pass-ios/clean/grounding/boarding-pass | 2 | success | Target hit | 2.21 s |
| flight-pass-ios/clean/grounding/check-in | 2 | success | Target hit | 2.25 s |
| flight-pass-ios/clean/grounding/trips-tab | 2 | success | Target hit | 1.87 s |
| flight-pass-ios/clean/grounding/profile-tab | 2 | success | Target hit | 2.24 s |
| flight-pass-ios/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; content is fully within the viewport, text fits its containers, and no elements overlap or obscure controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; content is fully within the viewport, text fits its containers, and no elements overlap or obscure controls." } | 2.20 s |
| flight-pass-ios/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 82 }, "description": "A red 'Flight update' banner overlaps the blue check-in button, hiding its label so the button text is cut off and the control is partially obscured." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 82 }, "description": "A red 'Flight update' banner overlaps the blue check-in button, hiding its label so the button text is cut off and the control is partially obscured." } | 2.08 s |
| flight-pass-ios/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.80 s |
| flight-pass-ios/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.71 s |
| smart-home-android/clean/grounding/more | 2 | success | Target hit | 1.91 s |
| smart-home-android/clean/grounding/room-search | 2 | success | Target hit | 1.75 s |
| smart-home-android/clean/grounding/thermostat-card | 2 | success | Target hit | 2.58 s |
| smart-home-android/clean/grounding/add-device | 2 | success | Target hit | 1.76 s |
| smart-home-android/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, elements are not overlapping, and nothing is cropped at the viewport edges." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, elements are not overlapping, and nothing is cropped at the viewport edges." } | 1.77 s |
| smart-home-android/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 531, "width": 628, "height": 55 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its chevron is cut off." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 531, "width": 628, "height": 55 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its chevron is cut off." } | 2.13 s |
| smart-home-android/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.74 s |
| smart-home-android/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.58 s |
| commerce-catalog-desktop/clean/grounding/search | 3 | success | Target hit | 2.58 s |
| commerce-catalog-desktop/clean/grounding/add-product | 3 | success | Target hit | 1.92 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 3 | success | Target hit | 3.00 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 3 | success | Target hit | 2.02 s |
| commerce-catalog-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact and aligned." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact and aligned." } | 2.56 s |
| commerce-catalog-desktop/defect/layout-defect | 3 | invalid response | The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 125, "width": 205, "height": 57 }, "description": "The page heading and eyebrow text are cut off at the left edge of their container. The eyebrow reads 'IANAGEMENT' instead of 'MANAGEMENT' and the title reads 'ducts' instead of 'Products'." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 125, "width": 205, "height": 57 }, "description": "The page heading and eyebrow text are cut off at the left edge of their container. The eyebrow reads 'IANAGEMENT' instead of 'MANAGEMENT' and the title reads 'ducts' instead of 'Products'." } | 3.32 s |
| commerce-catalog-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.68 s |
| commerce-catalog-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.86 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 3 | success | Target hit | 2.65 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 3 | success | Target hit | 1.85 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 3 | success | Target hit | 2.33 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 3 | success | Target hit | 2.31 s |
| analytics-dashboard-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, nothing is cropped or overlapping, and controls are unobstructed." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, nothing is cropped or overlapping, and controls are unobstructed." } | 2.16 s |
| analytics-dashboard-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 199, "y": 243, "width": 347, "height": 50 }, "description": "The red heading 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is truncated by the container." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 199, "y": 243, "width": 347, "height": 50 }, "description": "The red heading 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is truncated by the container." } | 2.14 s |
| analytics-dashboard-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.05 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.21 s |
| iot-control-desktop/clean/grounding/devices-nav | 3 | success | Target hit | 2.43 s |
| iot-control-desktop/clean/grounding/restart-device | 3 | success | Target hit | 2.00 s |
| iot-control-desktop/clean/grounding/auto-mode | 3 | success | Target hit | 2.25 s |
| iot-control-desktop/clean/grounding/pump-card | 3 | success | Target hit | 1.91 s |
| iot-control-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." } | 2.08 s |
| iot-control-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' banner floats over the table rows, covering part of the Cooling pump and Intake valve rows and breaking the table layout." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' banner floats over the table rows, covering part of the Cooling pump and Intake valve rows and breaking the table layout." } | 2.50 s |
| iot-control-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.88 s |
| iot-control-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.83 s |
| banking-ios/clean/grounding/profile | 3 | success | Target hit | 1.75 s |
| banking-ios/clean/grounding/transaction-search | 3 | success | Target hit | 2.71 s |
| banking-ios/clean/grounding/hide-balance | 3 | success | Target hit | 1.76 s |
| banking-ios/clean/grounding/transfer | 3 | success | Target hit | 1.72 s |
| banking-ios/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: text is fully contained, elements are aligned without overlap, and all controls are visible and unobstructed." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: text is fully contained, elements are aligned without overlap, and all controls are visible and unobstructed." } | 1.81 s |
| banking-ios/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 290, "height": 50 }, "description": "The header title 'Banking' and the brand label above it are cut off at the left edge of the screen, showing 'anking' and a partial 'MAYA' label." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 290, "height": 50 }, "description": "The header title 'Banking' and the brand label above it are cut off at the left edge of the screen, showing 'anking' and a partial 'MAYA' label." } | 1.91 s |
| banking-ios/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.70 s |
| banking-ios/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.92 s |
| flight-pass-ios/clean/grounding/boarding-pass | 3 | success | Target hit | 2.36 s |
| flight-pass-ios/clean/grounding/check-in | 3 | success | Target hit | 2.25 s |
| flight-pass-ios/clean/grounding/trips-tab | 3 | invalid response | Target missed | 2.21 s |
| flight-pass-ios/clean/grounding/profile-tab | 3 | success | Target hit | 1.73 s |
| flight-pass-ios/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: text, cards, button and tab bar are fully visible, aligned and not overlapping or clipped." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: text, cards, button and tab bar are fully visible, aligned and not overlapping or clipped." } | 2.12 s |
| flight-pass-ios/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 80 }, "description": "A red 'Flight update' banner overlaps the blue check-in button, covering most of its label so the button text is cut off and the control is partly hidden." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 80 }, "description": "A red 'Flight update' banner overlaps the blue check-in button, covering most of its label so the button text is cut off and the control is partly hidden." } | 2.08 s |
| flight-pass-ios/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.84 s |
| flight-pass-ios/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.54 s |
| smart-home-android/clean/grounding/more | 3 | success | Target hit | 1.58 s |
| smart-home-android/clean/grounding/room-search | 3 | success | Target hit | 1.86 s |
| smart-home-android/clean/grounding/thermostat-card | 3 | success | Target hit | 2.24 s |
| smart-home-android/clean/grounding/add-device | 3 | success | Target hit | 1.75 s |
| smart-home-android/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, elements do not overlap, and controls are fully visible and unobstructed." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, elements do not overlap, and controls are fully visible and unobstructed." } | 1.97 s |
| smart-home-android/defect/layout-defect | 3 | invalid response | The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 525, "width": 636, "height": 96 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its right side is cut off." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 525, "width": 636, "height": 96 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its right side is cut off." } | 2.42 s |
| smart-home-android/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.67 s |
| smart-home-android/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.53 s |
Gemini 3.8 Flash · 144 case executions
Returned model versions: google/gemini-3.8-flash.
Targets with the same hit/miss outcome across all three repetitions: 24 / 24. Agreement can include consistently missing a target.
| Case | Repeat | Result | Outcome | Time |
|---|---|---|---|---|
| commerce-catalog-desktop/clean/grounding/search | 1 | success | Target hit | 2.28 s |
| commerce-catalog-desktop/clean/grounding/add-product | 1 | success | Target hit | 3.52 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 1 | success | Target hit | 2.57 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 1 | success | Target hit | 2.31 s |
| commerce-catalog-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 3.65 s |
| commerce-catalog-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 67, "height": 68 }, "description": "The page heading text is clipped on the left edge, cutting off 'Products' into 'ducts' and 'MANAGEMENT' into 'IANAGEMENT'." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 67, "height": 68 }, "description": "The page heading text is clipped on the left edge, cutting off 'Products' into 'ducts' and 'MANAGEMENT' into 'IANAGEMENT'." } | 4.67 s |
| commerce-catalog-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.46 s |
| commerce-catalog-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 3.09 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 1 | success | Target hit | 2.08 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 1 | success | Target hit | 2.17 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 1 | success | Target hit | 2.41 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 1 | success | Target hit | 2.25 s |
| analytics-dashboard-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering issues detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering issues detected." } | 6.43 s |
| analytics-dashboard-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 202, "y": 251, "width": 326, "height": 28 }, "description": "The card heading text is clipped at the right boundary of its parent container." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 202, "y": 251, "width": 326, "height": 28 }, "description": "The card heading text is clipped at the right boundary of its parent container." } | 4.58 s |
| analytics-dashboard-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.44 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.84 s |
| iot-control-desktop/clean/grounding/devices-nav | 1 | success | Target hit | 2.46 s |
| iot-control-desktop/clean/grounding/restart-device | 1 | success | Target hit | 2.57 s |
| iot-control-desktop/clean/grounding/auto-mode | 1 | success | Target hit | 2.08 s |
| iot-control-desktop/clean/grounding/pump-card | 1 | success | Target hit | 2.26 s |
| iot-control-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering defects detected; elements are well-aligned, legible, and unclipped." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering defects detected; elements are well-aligned, legible, and unclipped." } | 12.60 s |
| iot-control-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The 'Maintenance scheduled' card overlaps the table content and separator lines." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The 'Maintenance scheduled' card overlaps the table content and separator lines." } | 4.28 s |
| iot-control-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.41 s |
| iot-control-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.22 s |
| banking-ios/clean/grounding/profile | 1 | success | Target hit | 10.16 s |
| banking-ios/clean/grounding/transaction-search | 1 | success | Target hit | 2.53 s |
| banking-ios/clean/grounding/hide-balance | 1 | success | Target hit | 1.82 s |
| banking-ios/clean/grounding/transfer | 1 | success | Target hit | 2.23 s |
| banking-ios/clean/layout-defect | 1 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 412, "y": 869, "width": 178, "height": 27 }, "description": "Two 'Transfer' text labels are vertically overlapping inside the button, creating a distorted, duplicated text rendering defect." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 412, "y": 869, "width": 178, "height": 27 }, "description": "Two 'Transfer' text labels are vertically overlapping inside the button, creating a distorted, duplicated text rendering defect." } | 13.11 s |
| banking-ios/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 59 }, "description": "The header text ('AYA' / 'banking') is cut off by the left viewport edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 59 }, "description": "The header text ('AYA' / 'banking') is cut off by the left viewport edge." } | 3.90 s |
| banking-ios/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.90 s |
| banking-ios/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.58 s |
| flight-pass-ios/clean/grounding/boarding-pass | 1 | success | Target hit | 2.34 s |
| flight-pass-ios/clean/grounding/check-in | 1 | success | Target hit | 2.16 s |
| flight-pass-ios/clean/grounding/trips-tab | 1 | success | Target hit | 1.65 s |
| flight-pass-ios/clean/grounding/profile-tab | 1 | success | Target hit | 1.71 s |
| flight-pass-ios/clean/layout-defect | 1 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 405, "y": 866, "width": 189, "height": 26 }, "description": "The \"Check in\" button text is clipped along its bottom edge inside the button container." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 405, "y": 866, "width": 189, "height": 26 }, "description": "The \"Check in\" button text is clipped along its bottom edge inside the button container." } | 12.77 s |
| flight-pass-ios/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 45, "y": 850, "width": 910, "height": 68 }, "description": "The red 'Flight update' banner overlaps and obscures the blue button beneath it." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 45, "y": 850, "width": 910, "height": 68 }, "description": "The red 'Flight update' banner overlaps and obscures the blue button beneath it." } | 4.16 s |
| flight-pass-ios/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.99 s |
| flight-pass-ios/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.89 s |
| smart-home-android/clean/grounding/more | 1 | success | Target hit | 2.05 s |
| smart-home-android/clean/grounding/room-search | 1 | success | Target hit | 2.36 s |
| smart-home-android/clean/grounding/thermostat-card | 1 | success | Target hit | 2.22 s |
| smart-home-android/clean/grounding/add-device | 1 | success | Target hit | 1.55 s |
| smart-home-android/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "None" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "None" } | 8.97 s |
| smart-home-android/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 355, "y": 510, "width": 600, "height": 69 }, "description": "The 'Kitchen lights' card is misaligned and unexpectedly indented/broken compared to the other list items in the 'Quick view' section." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 355, "y": 510, "width": 600, "height": 69 }, "description": "The 'Kitchen lights' card is misaligned and unexpectedly indented/broken compared to the other list items in the 'Quick view' section." } | 7.27 s |
| smart-home-android/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.35 s |
| smart-home-android/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.06 s |
| commerce-catalog-desktop/clean/grounding/search | 2 | success | Target hit | 2.53 s |
| commerce-catalog-desktop/clean/grounding/add-product | 2 | success | Target hit | 2.38 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 2 | success | Target hit | 2.62 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 2 | success | Target hit | 2.50 s |
| commerce-catalog-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects detected in the interface." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects detected in the interface." } | 4.11 s |
| commerce-catalog-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 68, "height": 73 }, "description": "The page header text ('Products' and category label above it) is horizontally clipped on the left edge by its container boundary." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 68, "height": 73 }, "description": "The page header text ('Products' and category label above it) is horizontally clipped on the left edge by its container boundary." } | 10.76 s |
| commerce-catalog-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.12 s |
| commerce-catalog-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 3.35 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 2 | success | Target hit | 8.75 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 2 | success | Target hit | 2.04 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 2 | success | Target hit | 1.87 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 2 | success | Target hit | 2.24 s |
| analytics-dashboard-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering defects detected." } | 5.95 s |
| analytics-dashboard-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 224, "width": 339, "height": 142 }, "description": "The heading text inside the first KPI card is cut off and clipped by the card's right boundary." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 224, "width": 339, "height": 142 }, "description": "The heading text inside the first KPI card is cut off and clipped by the card's right boundary." } | 3.10 s |
| analytics-dashboard-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.67 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.94 s |
| iot-control-desktop/clean/grounding/devices-nav | 2 | success | Target hit | 2.15 s |
| iot-control-desktop/clean/grounding/restart-device | 2 | success | Target hit | 2.22 s |
| iot-control-desktop/clean/grounding/auto-mode | 2 | success | Target hit | 2.25 s |
| iot-control-desktop/clean/grounding/pump-card | 2 | success | Target hit | 2.36 s |
| iot-control-desktop/clean/layout-defect | 2 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 185, "y": 118, "width": 227, "height": 70 }, "description": "The subtitle 'FACTORY FLOOR 02' overlaps directly onto the top of the main 'IoT control center' heading." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 185, "y": 118, "width": 227, "height": 70 }, "description": "The subtitle 'FACTORY FLOOR 02' overlaps directly onto the top of the main 'IoT control center' heading." } | 4.70 s |
| iot-control-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' badge or overlay is floating awkwardly and overlapping table row divider lines and table contents." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' badge or overlay is floating awkwardly and overlapping table row divider lines and table contents." } | 2.14 s |
| iot-control-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.26 s |
| iot-control-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.55 s |
| banking-ios/clean/grounding/profile | 2 | success | Target hit | 5.36 s |
| banking-ios/clean/grounding/transaction-search | 2 | success | Target hit | 2.49 s |
| banking-ios/clean/grounding/hide-balance | 2 | success | Target hit | 2.10 s |
| banking-ios/clean/grounding/transfer | 2 | success | Target hit | 2.07 s |
| banking-ios/clean/layout-defect | 2 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 413, "y": 868, "width": 174, "height": 22 }, "description": "The 'Transfer' button label has duplicated text overlapping vertically, rendering the text distorted and corrupted." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 413, "y": 868, "width": 174, "height": 22 }, "description": "The 'Transfer' button label has duplicated text overlapping vertically, rendering the text distorted and corrupted." } | 7.07 s |
| banking-ios/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 60 }, "description": "The header text in the upper left corner is clipped off by the left viewport edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 60 }, "description": "The header text in the upper left corner is clipped off by the left viewport edge." } | 4.05 s |
| banking-ios/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.54 s |
| banking-ios/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.07 s |
| flight-pass-ios/clean/grounding/boarding-pass | 2 | success | Target hit | 2.06 s |
| flight-pass-ios/clean/grounding/check-in | 2 | success | Target hit | 1.79 s |
| flight-pass-ios/clean/grounding/trips-tab | 2 | success | Target hit | 2.85 s |
| flight-pass-ios/clean/grounding/profile-tab | 2 | success | Target hit | 2.04 s |
| flight-pass-ios/clean/layout-defect | 2 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 852, "width": 908, "height": 59 }, "description": "The 'Check in' label inside the primary action button is vertically clipped at the bottom." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 852, "width": 908, "height": 59 }, "description": "The 'Check in' label inside the primary action button is vertically clipped at the bottom." } | 7.45 s |
| flight-pass-ios/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 46, "y": 849, "width": 908, "height": 67 }, "description": "The red 'Flight update' notification banner overlaps and obscures the blue button behind it." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 46, "y": 849, "width": 908, "height": 67 }, "description": "The red 'Flight update' notification banner overlaps and obscures the blue button behind it." } | 2.35 s |
| flight-pass-ios/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.73 s |
| flight-pass-ios/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.85 s |
| smart-home-android/clean/grounding/more | 2 | success | Target hit | 2.12 s |
| smart-home-android/clean/grounding/room-search | 2 | success | Target hit | 2.18 s |
| smart-home-android/clean/grounding/thermostat-card | 2 | success | Target hit | 2.07 s |
| smart-home-android/clean/grounding/add-device | 2 | success | Target hit | 2.02 s |
| smart-home-android/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No rendering defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No rendering defects detected." } | 6.38 s |
| smart-home-android/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 509, "width": 600, "height": 71 }, "description": "The second list card ('Kitchen lights') is severely misaligned horizontally compared to the other list items, breaking the expected vertical list layout." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 509, "width": 600, "height": 71 }, "description": "The second list card ('Kitchen lights') is severely misaligned horizontally compared to the other list items, breaking the expected vertical list layout." } | 8.71 s |
| smart-home-android/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.21 s |
| smart-home-android/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.38 s |
| commerce-catalog-desktop/clean/grounding/search | 3 | success | Target hit | 2.32 s |
| commerce-catalog-desktop/clean/grounding/add-product | 3 | success | Target hit | 2.40 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 3 | success | Target hit | 2.62 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 3 | success | Target hit | 3.04 s |
| commerce-catalog-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible defects detected." } | 10.99 s |
| commerce-catalog-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 127, "width": 68, "height": 67 }, "description": "The section title text ('ducts') and subtitle ('IANAGEMENT') are clipped on the left edge by the sidebar/container boundary." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 127, "width": 68, "height": 67 }, "description": "The section title text ('ducts') and subtitle ('IANAGEMENT') are clipped on the left edge by the sidebar/container boundary." } | 2.32 s |
| commerce-catalog-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.51 s |
| commerce-catalog-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.85 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 3 | success | Target hit | 9.88 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 3 | success | Target hit | 2.26 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 3 | success | Target hit | 2.06 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 3 | success | Target hit | 2.07 s |
| analytics-dashboard-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual defects detected in the interface." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual defects detected in the interface." } | 5.59 s |
| analytics-dashboard-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 187, "y": 223, "width": 340, "height": 143 }, "description": "The metric card title 'Monthly revenue across all sales chann' is clipped by the container's right boundary." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 187, "y": 223, "width": 340, "height": 143 }, "description": "The metric card title 'Monthly revenue across all sales chann' is clipped by the container's right boundary." } | 5.54 s |
| analytics-dashboard-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.80 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.89 s |
| iot-control-desktop/clean/grounding/devices-nav | 3 | success | Target hit | 2.31 s |
| iot-control-desktop/clean/grounding/restart-device | 3 | success | Target hit | 2.36 s |
| iot-control-desktop/clean/grounding/auto-mode | 3 | success | Target hit | 2.29 s |
| iot-control-desktop/clean/grounding/pump-card | 3 | success | Target hit | 2.74 s |
| iot-control-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects detected in the interface." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects detected in the interface." } | 11.74 s |
| iot-control-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 223, "height": 67 }, "description": "The 'Maintenance scheduled' banner overlaps directly on top of the data table and its row borders." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 223, "height": 67 }, "description": "The 'Maintenance scheduled' banner overlaps directly on top of the data table and its row borders." } | 13.83 s |
| iot-control-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.37 s |
| iot-control-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.58 s |
| banking-ios/clean/grounding/profile | 3 | success | Target hit | 12.22 s |
| banking-ios/clean/grounding/transaction-search | 3 | success | Target hit | 3.27 s |
| banking-ios/clean/grounding/hide-balance | 3 | success | Target hit | 2.09 s |
| banking-ios/clean/grounding/transfer | 3 | success | Target hit | 1.99 s |
| banking-ios/clean/layout-defect | 3 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 410, "y": 866, "width": 180, "height": 25 }, "description": "Two text labels within the 'Transfer' button overlap directly on top of each other, creating garbled and obscured text." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 410, "y": 866, "width": 180, "height": 25 }, "description": "Two text labels within the 'Transfer' button overlap directly on top of each other, creating garbled and obscured text." } | 8.34 s |
| banking-ios/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 62 }, "description": "The title text 'banking' and the header label above it are cropped by the left edge of the viewport." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 62 }, "description": "The title text 'banking' and the header label above it are cropped by the left edge of the viewport." } | 3.69 s |
| banking-ios/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.23 s |
| banking-ios/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.50 s |
| flight-pass-ios/clean/grounding/boarding-pass | 3 | success | Target hit | 2.05 s |
| flight-pass-ios/clean/grounding/check-in | 3 | success | Target hit | 2.25 s |
| flight-pass-ios/clean/grounding/trips-tab | 3 | success | Target hit | 2.51 s |
| flight-pass-ios/clean/grounding/profile-tab | 3 | success | Target hit | 1.92 s |
| flight-pass-ios/clean/layout-defect | 3 | invalid response | The answer is not valid JSON. Parsing failed before schema validation could run. JSON parsing error: Unterminated string in JSON at position 213 (line 1 column 214) Malformed answers are not repaired in this pilot and count as incorrect, even when the intended defect category is recognizable. Expected: { "has_issue": false, "category": null, "region": null } Returned: {"has_issue": true, "category": "container_clipping", "region": {"x": 409, "y": 869, "width": 182, "height": 22}, "description": "The bottom portion of the 'Check in' button text is clipped horizontally inside its · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: {"has_issue": true, "category": "container_clipping", "region": {"x": 409, "y": 869, "width": 182, "height": 22}, "description": "The bottom portion of the 'Check in' button text is clipped horizontally inside its | 17.47 s |
| flight-pass-ios/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 66 }, "description": "A red 'Flight update' notification banner overlaps and obscures the blue 'Check in' action button." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 66 }, "description": "A red 'Flight update' notification banner overlaps and obscures the blue 'Check in' action button." } | 4.31 s |
| flight-pass-ios/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.41 s |
| flight-pass-ios/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.49 s |
| smart-home-android/clean/grounding/more | 3 | success | Target hit | 1.79 s |
| smart-home-android/clean/grounding/room-search | 3 | success | Target hit | 2.07 s |
| smart-home-android/clean/grounding/thermostat-card | 3 | success | Target hit | 2.30 s |
| smart-home-android/clean/grounding/add-device | 3 | success | Target hit | 2.25 s |
| smart-home-android/clean/layout-defect | 3 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 140, "y": 966, "width": 725, "height": 34 }, "description": "The bottom navigation bar labels ('Home', 'Rooms', 'Activity') are partially cut off by the bottom viewport edge." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 140, "y": 966, "width": 725, "height": 34 }, "description": "The bottom navigation bar labels ('Home', 'Rooms', 'Activity') are partially cut off by the bottom viewport edge." } | 8.35 s |
| smart-home-android/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 355, "y": 510, "width": 600, "height": 70 }, "description": "The 'Kitchen lights' list item card is significantly shifted and misaligned from the vertical grid layout used by surrounding items." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 355, "y": 510, "width": 600, "height": 70 }, "description": "The 'Kitchen lights' list item card is significantly shifted and misaligned from the vertical grid layout used by surrounding items." } | 8.83 s |
| smart-home-android/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.84 s |
| smart-home-android/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.62 s |
Grok 4.7 · 144 case executions
Returned model versions: x-ai/grok-4.7.
Targets with the same hit/miss outcome across all three repetitions: 16 / 24. Agreement can include consistently missing a target.
| Case | Repeat | Result | Outcome | Time |
|---|---|---|---|---|
| commerce-catalog-desktop/clean/grounding/search | 1 | success | Target hit | 12.82 s |
| commerce-catalog-desktop/clean/grounding/add-product | 1 | success | Target hit | 7.11 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 1 | success | Target hit | 9.00 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 1 | success | Target hit | 28.50 s |
| commerce-catalog-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect." } | 12.36 s |
| commerce-catalog-desktop/defect/layout-defect | 1 | invalid response | The answer is not valid JSON. Parsing failed before schema validation could run. JSON parsing error: Unterminated string in JSON at position 195 (line 1 column 196) Malformed answers are not repaired in this pilot and count as incorrect, even when the intended defect category is recognizable. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: {"has_issue":true,"category":"container_clipping","region":{"x":278,"y":114,"width":99,"height":65},"description":"The page heading is clipped on the left, showing only “IANAGEMENT” and “ducts.”} · The answer is not valid JSON. Parsing failed before schema validation could run. JSON parsing error: Unterminated string in JSON at position 195 (line 1 column 196) Malformed answers are not repaired in this pilot and count as incorrect, even when the intended defect category is recognizable. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: {"has_issue":true,"category":"container_clipping","region":{"x":278,"y":114,"width":99,"height":65},"description":"The page heading is clipped on the left, showing only “IANAGEMENT” and “ducts.”} | 3.06 s |
| commerce-catalog-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.39 s |
| commerce-catalog-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.97 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 1 | success | Target hit | 26.90 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 1 | success | Target hit | 9.62 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 1 | success | Target hit | 11.49 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 1 | success | Target missed | 27.51 s |
| analytics-dashboard-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, overflow, overlap, obscured control, or broken reflow is visible." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, overflow, overlap, obscured control, or broken reflow is visible." } | 92.19 s |
| analytics-dashboard-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 204, "y": 226, "width": 318, "height": 34 }, "description": "The revenue card title is truncated mid-word at the card's right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 204, "y": 226, "width": 318, "height": 34 }, "description": "The revenue card title is truncated mid-word at the card's right edge." } | 18.34 s |
| analytics-dashboard-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.54 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 18.39 s |
| iot-control-desktop/clean/grounding/devices-nav | 1 | success | Target hit | 13.27 s |
| iot-control-desktop/clean/grounding/restart-device | 1 | success | Target hit | 15.28 s |
| iot-control-desktop/clean/grounding/auto-mode | 1 | success | Target hit | 25.79 s |
| iot-control-desktop/clean/grounding/pump-card | 1 | success | Target hit | 14.44 s |
| iot-control-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect; all text, controls, cards, and table rows are fully contained and aligned." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect; all text, controls, cards, and table rows are fully contained and aligned." } | 6.73 s |
| iot-control-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 556, "width": 219, "height": 66 }, "description": "A red 'Maintenance scheduled' banner overlaps the device table rows." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 556, "width": 219, "height": 66 }, "description": "A red 'Maintenance scheduled' banner overlaps the device table rows." } | 25.49 s |
| iot-control-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.88 s |
| iot-control-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.87 s |
| banking-ios/clean/grounding/profile | 1 | success | Target missed | 21.93 s |
| banking-ios/clean/grounding/transaction-search | 1 | success | Target hit | 10.19 s |
| banking-ios/clean/grounding/hide-balance | 1 | success | Target hit | 54.00 s |
| banking-ios/clean/grounding/transfer | 1 | success | Target hit | 11.31 s |
| banking-ios/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the listed categories." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the listed categories." } | 8.19 s |
| banking-ios/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 224, "height": 34 }, "description": "The 'Banking' heading is cropped by the left viewport edge, leaving only 'anking' visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 224, "height": 34 }, "description": "The 'Banking' heading is cropped by the left viewport edge, leaving only 'anking' visible." } | 20.63 s |
| banking-ios/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.66 s |
| banking-ios/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.00 s |
| flight-pass-ios/clean/grounding/boarding-pass | 1 | success | Target hit | 14.68 s |
| flight-pass-ios/clean/grounding/check-in | 1 | success | Target hit | 8.91 s |
| flight-pass-ios/clean/grounding/trips-tab | 1 | success | Target hit | 17.82 s |
| flight-pass-ios/clean/grounding/profile-tab | 1 | success | Target hit | 17.09 s |
| flight-pass-ios/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 12.00 s |
| flight-pass-ios/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 48, "y": 842, "width": 904, "height": 90 }, "description": "Red Flight update banner overlaps a blue banner, obscuring its text." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 48, "y": 842, "width": 904, "height": 90 }, "description": "Red Flight update banner overlaps a blue banner, obscuring its text." } | 19.37 s |
| flight-pass-ios/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.86 s |
| flight-pass-ios/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.58 s |
| smart-home-android/clean/grounding/more | 1 | success | Target missed | 8.75 s |
| smart-home-android/clean/grounding/room-search | 1 | success | Target hit | 9.33 s |
| smart-home-android/clean/grounding/thermostat-card | 1 | success | Target hit | 13.72 s |
| smart-home-android/clean/grounding/add-device | 1 | success | Target missed | 8.84 s |
| smart-home-android/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 3.17 s |
| smart-home-android/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 366, "y": 509, "width": 588, "height": 83 }, "description": "The Kitchen lights card is shifted right and does not align with the full-width cards above and below it." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 366, "y": 509, "width": 588, "height": 83 }, "description": "The Kitchen lights card is shifted right and does not align with the full-width cards above and below it." } | 20.28 s |
| smart-home-android/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.88 s |
| smart-home-android/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.52 s |
| commerce-catalog-desktop/clean/grounding/search | 2 | success | Target hit | 26.80 s |
| commerce-catalog-desktop/clean/grounding/add-product | 2 | success | Target hit | 7.36 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 2 | success | Target hit | 20.98 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 2 | success | Target missed | 17.11 s |
| commerce-catalog-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 5.63 s |
| commerce-catalog-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 276, "y": 112, "width": 103, "height": 68 }, "description": "The page heading is clipped at its left edge, cutting off the beginning of both heading lines." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 276, "y": 112, "width": 103, "height": 68 }, "description": "The page heading is clipped at its left edge, cutting off the beginning of both heading lines." } | 3.07 s |
| commerce-catalog-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.73 s |
| commerce-catalog-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.08 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 2 | success | Target hit | 18.38 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 2 | success | Target hit | 16.61 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 2 | success | Target missed | 38.77 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 2 | success | Target missed | 21.24 s |
| analytics-dashboard-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect." } | 45.43 s |
| analytics-dashboard-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 228, "width": 324, "height": 46 }, "description": "The metric card heading is cut off mid-word at the card's right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 228, "width": 324, "height": 46 }, "description": "The metric card heading is cut off mid-word at the card's right edge." } | 14.96 s |
| analytics-dashboard-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.22 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.97 s |
| iot-control-desktop/clean/grounding/devices-nav | 2 | success | Target hit | 21.10 s |
| iot-control-desktop/clean/grounding/restart-device | 2 | success | Target hit | 11.53 s |
| iot-control-desktop/clean/grounding/auto-mode | 2 | success | Target missed | 16.62 s |
| iot-control-desktop/clean/grounding/pump-card | 2 | success | Target hit | 22.31 s |
| iot-control-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects." } | 13.76 s |
| iot-control-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 458, "y": 555, "width": 224, "height": 69 }, "description": "A red \"Maintenance scheduled\" banner overlaps and obscures the device table rows." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 458, "y": 555, "width": 224, "height": 69 }, "description": "A red \"Maintenance scheduled\" banner overlaps and obscures the device table rows." } | 12.82 s |
| iot-control-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.78 s |
| iot-control-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.89 s |
| banking-ios/clean/grounding/profile | 2 | success | Target missed | 20.19 s |
| banking-ios/clean/grounding/transaction-search | 2 | success | Target hit | 24.96 s |
| banking-ios/clean/grounding/hide-balance | 2 | success | Target missed | 8.56 s |
| banking-ios/clean/grounding/transfer | 2 | success | Target hit | 8.18 s |
| banking-ios/clean/layout-defect | 2 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 58, "y": 962, "width": 814, "height": 38 }, "description": "Bottom navigation labels are cut off by the lower viewport edge." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 58, "y": 962, "width": 814, "height": 38 }, "description": "Bottom navigation labels are cut off by the lower viewport edge." } | 18.85 s |
| banking-ios/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 58, "width": 500, "height": 48 }, "description": "The 'Banking' heading is cropped at the left viewport edge, with the leading 'B' cut off." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 58, "width": 500, "height": 48 }, "description": "The 'Banking' heading is cropped at the left viewport edge, with the leading 'B' cut off." } | 20.32 s |
| banking-ios/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.54 s |
| banking-ios/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.83 s |
| flight-pass-ios/clean/grounding/boarding-pass | 2 | success | Target hit | 12.05 s |
| flight-pass-ios/clean/grounding/check-in | 2 | success | Target missed | 15.24 s |
| flight-pass-ios/clean/grounding/trips-tab | 2 | success | Target hit | 23.46 s |
| flight-pass-ios/clean/grounding/profile-tab | 2 | success | Target hit | 25.79 s |
| flight-pass-ios/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect; content fits the viewport without clipping, overflow, overlap, or broken reflow." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect; content fits the viewport without clipping, overflow, overlap, or broken reflow." } | 7.64 s |
| flight-pass-ios/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 42, "y": 812, "width": 918, "height": 92 }, "description": "A red flight-update banner overlaps a blue card behind it, obscuring that card's text." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 42, "y": 812, "width": 918, "height": 92 }, "description": "A red flight-update banner overlaps a blue card behind it, obscuring that card's text." } | 12.45 s |
| flight-pass-ios/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.96 s |
| flight-pass-ios/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.77 s |
| smart-home-android/clean/grounding/more | 2 | success | Target missed | 16.25 s |
| smart-home-android/clean/grounding/room-search | 2 | success | Target hit | 16.47 s |
| smart-home-android/clean/grounding/thermostat-card | 2 | success | Target hit | 19.16 s |
| smart-home-android/clean/grounding/add-device | 2 | success | Target hit | 9.97 s |
| smart-home-android/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect." } | 15.40 s |
| smart-home-android/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 328, "y": 567, "width": 632, "height": 110 }, "description": "The Kitchen lights card is shifted right and overlaps the Front door card beneath it." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 328, "y": 567, "width": 632, "height": 110 }, "description": "The Kitchen lights card is shifted right and overlaps the Front door card beneath it." } | 40.90 s |
| smart-home-android/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.51 s |
| smart-home-android/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 5.93 s |
| commerce-catalog-desktop/clean/grounding/search | 3 | success | Target hit | 27.72 s |
| commerce-catalog-desktop/clean/grounding/add-product | 3 | invalid response | Target missed | 11.21 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 3 | success | Target hit | 21.57 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 3 | success | Target missed | 17.41 s |
| commerce-catalog-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, overflow, overlap, obscured control, or broken reflow is visible." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, overflow, overlap, obscured control, or broken reflow is visible." } | 13.76 s |
| commerce-catalog-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 273, "y": 116, "width": 102, "height": 62 }, "description": "Heading text is clipped at its left edge, showing incomplete words." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 273, "y": 116, "width": 102, "height": 62 }, "description": "Heading text is clipped at its left edge, showing incomplete words." } | 3.93 s |
| commerce-catalog-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.19 s |
| commerce-catalog-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.49 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 3 | success | Target hit | 24.18 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 3 | success | Target hit | 9.67 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 3 | success | Target hit | 6.95 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 3 | success | Target missed | 10.42 s |
| analytics-dashboard-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the listed categories." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the listed categories." } | 35.63 s |
| analytics-dashboard-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 294, "y": 224, "width": 459, "height": 34 }, "description": "The revenue card heading is cut off at the card's right edge, truncating the title text." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 294, "y": 224, "width": 459, "height": 34 }, "description": "The revenue card heading is cut off at the card's right edge, truncating the title text." } | 18.82 s |
| analytics-dashboard-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 6.03 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.56 s |
| iot-control-desktop/clean/grounding/devices-nav | 3 | success | Target hit | 13.95 s |
| iot-control-desktop/clean/grounding/restart-device | 3 | success | Target hit | 8.91 s |
| iot-control-desktop/clean/grounding/auto-mode | 3 | success | Target hit | 18.87 s |
| iot-control-desktop/clean/grounding/pump-card | 3 | success | Target hit | 19.79 s |
| iot-control-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect; text, cards, table, and controls render within their containers." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect; text, cards, table, and controls render within their containers." } | 13.84 s |
| iot-control-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 463, "y": 556, "width": 223, "height": 68 }, "description": "A red maintenance banner overlaps the device table, covering the Cooling pump status and row boundary." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 463, "y": 556, "width": 223, "height": 68 }, "description": "A red maintenance banner overlaps the device table, covering the Cooling pump status and row boundary." } | 5.51 s |
| iot-control-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.98 s |
| iot-control-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.08 s |
| banking-ios/clean/grounding/profile | 3 | success | Target missed | 17.73 s |
| banking-ios/clean/grounding/transaction-search | 3 | success | Target hit | 17.27 s |
| banking-ios/clean/grounding/hide-balance | 3 | success | Target missed | 21.90 s |
| banking-ios/clean/grounding/transfer | 3 | success | Target hit | 7.91 s |
| banking-ios/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 9.43 s |
| banking-ios/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 49, "width": 162, "height": 44 }, "description": "The Banking header title is cropped at the left viewport edge, leaving only anking visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 49, "width": 162, "height": 44 }, "description": "The Banking header title is cropped at the left viewport edge, leaving only anking visible." } | 12.40 s |
| banking-ios/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.41 s |
| banking-ios/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.02 s |
| flight-pass-ios/clean/grounding/boarding-pass | 3 | success | Target hit | 12.12 s |
| flight-pass-ios/clean/grounding/check-in | 3 | success | Target missed | 17.61 s |
| flight-pass-ios/clean/grounding/trips-tab | 3 | success | Target hit | 18.95 s |
| flight-pass-ios/clean/grounding/profile-tab | 3 | success | Target hit | 32.81 s |
| flight-pass-ios/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect in the specified categories." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect in the specified categories." } | 10.62 s |
| flight-pass-ios/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 41, "y": 829, "width": 877, "height": 90 }, "description": "The red Flight update banner overlaps the blue banner behind it, obscuring and clipping its text." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 41, "y": 829, "width": 877, "height": 90 }, "description": "The red Flight update banner overlaps the blue banner behind it, obscuring and clipping its text." } | 17.93 s |
| flight-pass-ios/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.46 s |
| flight-pass-ios/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.66 s |
| smart-home-android/clean/grounding/more | 3 | success | Target missed | 4.33 s |
| smart-home-android/clean/grounding/room-search | 3 | success | Target hit | 6.48 s |
| smart-home-android/clean/grounding/thermostat-card | 3 | success | Target missed | 22.65 s |
| smart-home-android/clean/grounding/add-device | 3 | success | Target hit | 9.77 s |
| smart-home-android/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the specified categories." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the specified categories." } | 10.69 s |
| smart-home-android/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 430, "y": 510, "width": 528, "height": 80 }, "description": "The Kitchen lights card is indented and narrower than the full-width sibling cards, breaking the list reflow." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 430, "y": 510, "width": 528, "height": 80 }, "description": "The Kitchen lights card is indented and narrower than the full-width sibling cards, breaking the list reflow." } | 23.64 s |
| smart-home-android/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.57 s |
| smart-home-android/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.55 s |
Qwen3.8 Max (0902) · 144 case executions
Returned model versions: qwen/qwen3.8-max-0902.
Targets with the same hit/miss outcome across all three repetitions: 24 / 24. Agreement can include consistently missing a target.
| Case | Repeat | Result | Outcome | Time |
|---|---|---|---|---|
| commerce-catalog-desktop/clean/grounding/search | 1 | success | Target hit | 6.70 s |
| commerce-catalog-desktop/clean/grounding/add-product | 1 | success | Target hit | 9.03 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 1 | success | Target hit | 9.96 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 1 | success | Target hit | 7.91 s |
| commerce-catalog-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } | 11.01 s |
| commerce-catalog-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 190, "y": 120, "width": 180, "height": 75 }, "description": "Page heading is cut off at the left edge, showing 'ducts' instead of 'Products' and 'IANAGEMENT' instead of 'MANAGEMENT'." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 190, "y": 120, "width": 180, "height": 75 }, "description": "Page heading is cut off at the left edge, showing 'ducts' instead of 'Products' and 'IANAGEMENT' instead of 'MANAGEMENT'." } | 15.27 s |
| commerce-catalog-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 4.24 s |
| commerce-catalog-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 5.16 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 1 | success | Target hit | 14.38 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 1 | success | Target hit | 11.24 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 1 | success | Target hit | 10.90 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 1 | success | Target hit | 11.42 s |
| analytics-dashboard-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } | 11.64 s |
| analytics-dashboard-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 200, "y": 245, "width": 330, "height": 40 }, "description": "Card title 'Monthly revenue across all sales channels' is cut off at the card's right edge, truncating the word 'channels'." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 200, "y": 245, "width": 330, "height": 40 }, "description": "Card title 'Monthly revenue across all sales channels' is cut off at the card's right edge, truncating the word 'channels'." } | 14.56 s |
| analytics-dashboard-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 4.22 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 5.52 s |
| iot-control-desktop/clean/grounding/devices-nav | 1 | success | Target hit | 11.78 s |
| iot-control-desktop/clean/grounding/restart-device | 1 | success | Target hit | 11.50 s |
| iot-control-desktop/clean/grounding/auto-mode | 1 | success | Target hit | 14.37 s |
| iot-control-desktop/clean/grounding/pump-card | 1 | success | Target hit | 8.43 s |
| iot-control-desktop/clean/layout-defect | 1 | invalid response | The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: cards, table, toggle, and button render fully within their containers without cropping, clipping, overflow, overlap, or obscured controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: cards, table, toggle, and button render fully within their containers without cropping, clipping, overflow, overlap, or obscured controls." } | 10.94 s |
| iot-control-desktop/defect/layout-defect | 1 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; toast notification overlays content intentionally without obscuring actionable controls." } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; toast notification overlays content intentionally without obscuring actionable controls." } | 14.07 s |
| iot-control-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.83 s |
| iot-control-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 6.18 s |
| banking-ios/clean/grounding/profile | 1 | success | Target hit | 9.52 s |
| banking-ios/clean/grounding/transaction-search | 1 | success | Target hit | 6.30 s |
| banking-ios/clean/grounding/hide-balance | 1 | success | Target hit | 18.69 s |
| banking-ios/clean/grounding/transfer | 1 | success | Target hit | 11.96 s |
| banking-ios/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all containers, text, and controls render fully within the viewport without clipping, overflow, or overlap." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all containers, text, and controls render fully within the viewport without clipping, overflow, or overlap." } | 9.44 s |
| banking-ios/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 280, "height": 60 }, "description": "Page heading 'Banking' is clipped at the left edge; the leading 'B' is cut off by the viewport." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 280, "height": 60 }, "description": "Page heading 'Banking' is clipped at the left edge; the leading 'B' is cut off by the viewport." } | 9.56 s |
| banking-ios/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.62 s |
| banking-ios/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 4.11 s |
| flight-pass-ios/clean/grounding/boarding-pass | 1 | success | Target hit | 8.08 s |
| flight-pass-ios/clean/grounding/check-in | 1 | success | Target hit | 6.42 s |
| flight-pass-ios/clean/grounding/trips-tab | 1 | success | Target hit | 8.58 s |
| flight-pass-ios/clean/grounding/profile-tab | 1 | success | Target hit | 7.65 s |
| flight-pass-ios/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } | 8.78 s |
| flight-pass-ios/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 855, "width": 920, "height": 65 }, "description": "Red 'Flight update' toast banner overlaps and obscures the blue 'Check in' button text beneath it." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 855, "width": 920, "height": 65 }, "description": "Red 'Flight update' toast banner overlaps and obscures the blue 'Check in' button text beneath it." } | 14.78 s |
| flight-pass-ios/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.73 s |
| flight-pass-ios/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 4.51 s |
| smart-home-android/clean/grounding/more | 1 | success | Target hit | 9.39 s |
| smart-home-android/clean/grounding/room-search | 1 | success | Target hit | 5.70 s |
| smart-home-android/clean/grounding/thermostat-card | 1 | success | Target hit | 7.16 s |
| smart-home-android/clean/grounding/add-device | 1 | success | Target hit | 6.43 s |
| smart-home-android/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without overlap or clipping." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without overlap or clipping." } | 9.54 s |
| smart-home-android/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 355, "y": 510, "width": 645, "height": 70 }, "description": "The 'Kitchen lights' card (item 2) is shifted right and clipped at the right viewport edge, cutting off its border and chevron control." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 355, "y": 510, "width": 645, "height": 70 }, "description": "The 'Kitchen lights' card (item 2) is shifted right and clipped at the right viewport edge, cutting off its border and chevron control." } | 14.85 s |
| smart-home-android/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.91 s |
| smart-home-android/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 4.46 s |
| commerce-catalog-desktop/clean/grounding/search | 2 | success | Target hit | 7.87 s |
| commerce-catalog-desktop/clean/grounding/add-product | 2 | success | Target hit | 10.41 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 2 | success | Target hit | 10.06 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 2 | success | Target hit | 12.58 s |
| commerce-catalog-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all content fits within containers and viewport." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all content fits within containers and viewport." } | 8.64 s |
| commerce-catalog-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 190, "y": 120, "width": 180, "height": 75 }, "description": "Page heading 'Products' is clipped on the left, showing 'ducts', and the 'MANAGEMENT' eyebrow label is cut off at the container edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 190, "y": 120, "width": 180, "height": 75 }, "description": "Page heading 'Products' is clipped on the left, showing 'ducts', and the 'MANAGEMENT' eyebrow label is cut off at the container edge." } | 13.24 s |
| commerce-catalog-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.67 s |
| commerce-catalog-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 5.86 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 2 | success | Target hit | 11.26 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 2 | success | Target hit | 9.96 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 2 | success | Target hit | 9.24 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 2 | success | Target hit | 12.36 s |
| analytics-dashboard-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all content fits within containers and viewport." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all content fits within containers and viewport." } | 10.56 s |
| analytics-dashboard-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 245, "width": 340, "height": 40 }, "description": "Card title 'Monthly revenue across all sales channels' is truncated at the right edge of its container, cutting off the final letters." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 245, "width": 340, "height": 40 }, "description": "Card title 'Monthly revenue across all sales channels' is truncated at the right edge of its container, cutting off the final letters." } | 13.44 s |
| analytics-dashboard-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 4.18 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 5.57 s |
| iot-control-desktop/clean/grounding/devices-nav | 2 | success | Target hit | 6.57 s |
| iot-control-desktop/clean/grounding/restart-device | 2 | success | Target hit | 8.10 s |
| iot-control-desktop/clean/grounding/auto-mode | 2 | success | Target hit | 10.17 s |
| iot-control-desktop/clean/grounding/pump-card | 2 | success | Target hit | 5.96 s |
| iot-control-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without overlap or clipping." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without overlap or clipping." } | 9.94 s |
| iot-control-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 223, "height": 63 }, "description": "A red 'Maintenance scheduled' toast/banner overlays the device table rows, covering the area between the Cooling pump and Intake valve entries." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 223, "height": 63 }, "description": "A red 'Maintenance scheduled' toast/banner overlays the device table rows, covering the area between the Cooling pump and Intake valve entries." } | 22.10 s |
| iot-control-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.75 s |
| iot-control-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 5.03 s |
| banking-ios/clean/grounding/profile | 2 | success | Target hit | 8.60 s |
| banking-ios/clean/grounding/transaction-search | 2 | success | Target hit | 8.13 s |
| banking-ios/clean/grounding/hide-balance | 2 | success | Target hit | 6.74 s |
| banking-ios/clean/grounding/transfer | 2 | success | Target hit | 6.15 s |
| banking-ios/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } | 8.60 s |
| banking-ios/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 300, "height": 60 }, "description": "Page title 'Banking' is cut off at the left viewport edge, showing only 'anking'." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 300, "height": 60 }, "description": "Page title 'Banking' is cut off at the left viewport edge, showing only 'anking'." } | 11.66 s |
| banking-ios/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.97 s |
| banking-ios/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 6.69 s |
| flight-pass-ios/clean/grounding/boarding-pass | 2 | success | Target hit | 10.94 s |
| flight-pass-ios/clean/grounding/check-in | 2 | success | Target hit | 6.76 s |
| flight-pass-ios/clean/grounding/trips-tab | 2 | success | Target hit | 7.43 s |
| flight-pass-ios/clean/grounding/profile-tab | 2 | success | Target hit | 7.58 s |
| flight-pass-ios/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all text fits, controls are unobscured, and elements are properly spaced." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all text fits, controls are unobscured, and elements are properly spaced." } | 9.35 s |
| flight-pass-ios/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 920, "height": 70 }, "description": "The red 'Flight update' toast banner overlaps and obscures the blue 'Check in' button text beneath it." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 920, "height": 70 }, "description": "The red 'Flight update' toast banner overlaps and obscures the blue 'Check in' button text beneath it." } | 11.78 s |
| flight-pass-ios/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.27 s |
| flight-pass-ios/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 5.39 s |
| smart-home-android/clean/grounding/more | 2 | success | Target hit | 6.06 s |
| smart-home-android/clean/grounding/room-search | 2 | success | Target hit | 6.43 s |
| smart-home-android/clean/grounding/thermostat-card | 2 | success | Target hit | 6.58 s |
| smart-home-android/clean/grounding/add-device | 2 | success | Target hit | 6.15 s |
| smart-home-android/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without cropping, clipping, overflow, overlap, or obscured controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without cropping, clipping, overflow, overlap, or obscured controls." } | 9.39 s |
| smart-home-android/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 350, "y": 510, "width": 650, "height": 75 }, "description": "The 'Kitchen lights' quick-view card is offset to the right and clipped by the right viewport edge, breaking the list's alignment." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 350, "y": 510, "width": 650, "height": 75 }, "description": "The 'Kitchen lights' quick-view card is offset to the right and clipped by the right viewport edge, breaking the list's alignment." } | 17.60 s |
| smart-home-android/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 4.70 s |
| smart-home-android/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 6.14 s |
| commerce-catalog-desktop/clean/grounding/search | 3 | success | Target hit | 9.29 s |
| commerce-catalog-desktop/clean/grounding/add-product | 3 | success | Target hit | 10.12 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 3 | success | Target hit | 9.57 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 3 | success | Target hit | 5.48 s |
| commerce-catalog-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within their containers and the viewport." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within their containers and the viewport." } | 7.69 s |
| commerce-catalog-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 165, "y": 110, "width": 120, "height": 70 }, "description": "Page heading is cut off on the left: 'MANAGEMENT' appears as 'IANAGEMENT' and 'Products' as 'ducts', clipped by the sidebar edge." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 165, "y": 110, "width": 120, "height": 70 }, "description": "Page heading is cut off on the left: 'MANAGEMENT' appears as 'IANAGEMENT' and 'Products' as 'ducts', clipped by the sidebar edge." } | 16.10 s |
| commerce-catalog-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 4.78 s |
| commerce-catalog-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 8.14 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 3 | success | Target hit | 7.93 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 3 | success | Target hit | 12.24 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 3 | success | Target hit | 15.80 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 3 | success | Target hit | 11.01 s |
| analytics-dashboard-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } | 10.27 s |
| analytics-dashboard-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 248, "width": 328, "height": 32 }, "description": "Card title \"Monthly revenue across all sales chann\" is truncated at the card's right edge, cutting off the word \"channels\"." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 248, "width": 328, "height": 32 }, "description": "Card title \"Monthly revenue across all sales chann\" is truncated at the card's right edge, cutting off the word \"channels\"." } | 12.46 s |
| analytics-dashboard-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 7.10 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 4.95 s |
| iot-control-desktop/clean/grounding/devices-nav | 3 | success | Target hit | 5.90 s |
| iot-control-desktop/clean/grounding/restart-device | 3 | success | Target hit | 8.96 s |
| iot-control-desktop/clean/grounding/auto-mode | 3 | success | Target hit | 7.81 s |
| iot-control-desktop/clean/grounding/pump-card | 3 | success | Target hit | 8.01 s |
| iot-control-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects; layout is intact with no cropping, clipping, overflow, overlap, or obscured controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects; layout is intact with no cropping, clipping, overflow, overlap, or obscured controls." } | 10.18 s |
| iot-control-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 222, "height": 65 }, "description": "A red 'Maintenance scheduled' toast/banner overlaps the device table rows, covering content between the Cooling pump and Intake valve rows." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 222, "height": 65 }, "description": "A red 'Maintenance scheduled' toast/banner overlaps the device table rows, covering content between the Cooling pump and Intake valve rows." } | 11.95 s |
| iot-control-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.77 s |
| iot-control-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 6.03 s |
| banking-ios/clean/grounding/profile | 3 | success | Target hit | 9.36 s |
| banking-ios/clean/grounding/transaction-search | 3 | success | Target hit | 6.13 s |
| banking-ios/clean/grounding/hide-balance | 3 | success | Target hit | 8.05 s |
| banking-ios/clean/grounding/transfer | 3 | success | Target hit | 6.87 s |
| banking-ios/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without cropping, clipping, overflow, overlap, or obscured controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without cropping, clipping, overflow, overlap, or obscured controls." } | 9.48 s |
| banking-ios/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 280, "height": 50 }, "description": "Page title 'Banking' is cut off at the left viewport edge, showing only 'anking'." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 280, "height": 50 }, "description": "Page title 'Banking' is cut off at the left viewport edge, showing only 'anking'." } | 9.28 s |
| banking-ios/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 4.25 s |
| banking-ios/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 4.52 s |
| flight-pass-ios/clean/grounding/boarding-pass | 3 | success | Target hit | 7.59 s |
| flight-pass-ios/clean/grounding/check-in | 3 | success | Target hit | 9.32 s |
| flight-pass-ios/clean/grounding/trips-tab | 3 | success | Target hit | 6.66 s |
| flight-pass-ios/clean/grounding/profile-tab | 3 | success | Target hit | 8.68 s |
| flight-pass-ios/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all text fits within containers, controls are unobscured, and elements do not overlap or clip." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all text fits within containers, controls are unobscured, and elements do not overlap or clip." } | 8.34 s |
| flight-pass-ios/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 45, "y": 850, "width": 910, "height": 70 }, "description": "A red 'Flight update' toast banner overlaps and obscures a blue action button ('Check in') near the bottom of the screen." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 45, "y": 850, "width": 910, "height": 70 }, "description": "A red 'Flight update' toast banner overlaps and obscures a blue action button ('Check in') near the bottom of the screen." } | 14.65 s |
| flight-pass-ios/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.98 s |
| flight-pass-ios/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 4.74 s |
| smart-home-android/clean/grounding/more | 3 | success | Target hit | 8.92 s |
| smart-home-android/clean/grounding/room-search | 3 | success | Target hit | 6.79 s |
| smart-home-android/clean/grounding/thermostat-card | 3 | success | Target hit | 7.32 s |
| smart-home-android/clean/grounding/add-device | 3 | success | Target hit | 6.50 s |
| smart-home-android/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } | 8.80 s |
| smart-home-android/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 355, "y": 508, "width": 645, "height": 70 }, "description": "The 'Kitchen lights' quick-view card is offset to the right and its right edge is clipped by the viewport, unlike the other aligned cards." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 355, "y": 508, "width": 645, "height": 70 }, "description": "The 'Kitchen lights' quick-view card is offset to the right and its right edge is clipped by the viewport, unlike the other aligned cards." } | 14.15 s |
| smart-home-android/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.64 s |
| smart-home-android/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 5.56 s |
Muse Spark 1.3 · 144 case executions
Returned model versions: meta/muse-spark-1.3.
Targets with the same hit/miss outcome across all three repetitions: 23 / 24. Agreement can include consistently missing a target.
| Case | Repeat | Result | Outcome | Time |
|---|---|---|---|---|
| commerce-catalog-desktop/clean/grounding/search | 1 | success | Target hit | 5.13 s |
| commerce-catalog-desktop/clean/grounding/add-product | 1 | success | Target hit | 2.39 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 1 | success | Target hit | 3.01 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 1 | success | Target hit | 6.57 s |
| commerce-catalog-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defects observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defects observed.", "has_issue": false, "region": null } | 4.09 s |
| commerce-catalog-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Product heading clipped on left edge showing 'IANAGEMENT' and 'ducts' instead of full title.", "has_issue": true, "region": { "height": 65, "width": 68, "x": 191, "y": 126 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Product heading clipped on left edge showing 'IANAGEMENT' and 'ducts' instead of full title.", "has_issue": true, "region": { "height": 65, "width": 68, "x": 191, "y": 126 } } | 7.80 s |
| commerce-catalog-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.66 s |
| commerce-catalog-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.79 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 1 | success | Target hit | 4.96 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 1 | success | Target hit | 5.08 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 1 | success | Target hit | 3.09 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 1 | success | Target hit | 4.94 s |
| analytics-dashboard-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } | 4.07 s |
| analytics-dashboard-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card title truncated to 'Monthly revenue across all sales chann', cutting off end of word.", "has_issue": true, "region": { "height": 30, "width": 330, "x": 200, "y": 248 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card title truncated to 'Monthly revenue across all sales chann', cutting off end of word.", "has_issue": true, "region": { "height": 30, "width": 330, "x": 200, "y": 248 } } | 8.22 s |
| analytics-dashboard-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.36 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.71 s |
| iot-control-desktop/clean/grounding/devices-nav | 1 | success | Target hit | 3.78 s |
| iot-control-desktop/clean/grounding/restart-device | 1 | success | Target hit | 3.54 s |
| iot-control-desktop/clean/grounding/auto-mode | 1 | success | Target hit | 6.48 s |
| iot-control-desktop/clean/grounding/pump-card | 1 | success | Target hit | 5.57 s |
| iot-control-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } | 8.45 s |
| iot-control-desktop/defect/layout-defect | 1 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } | 5.07 s |
| iot-control-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.95 s |
| iot-control-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.71 s |
| banking-ios/clean/grounding/profile | 1 | success | Target hit | 6.58 s |
| banking-ios/clean/grounding/transaction-search | 1 | success | Target hit | 3.56 s |
| banking-ios/clean/grounding/hide-balance | 1 | success | Target missed | 8.77 s |
| banking-ios/clean/grounding/transfer | 1 | success | Target hit | 3.70 s |
| banking-ios/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } | 4.67 s |
| banking-ios/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "Header title 'banking' is cropped at left viewport edge with first letter partially cut off.", "has_issue": true, "region": { "height": 60, "width": 260, "x": 35, "y": 58 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "Header title 'banking' is cropped at left viewport edge with first letter partially cut off.", "has_issue": true, "region": { "height": 60, "width": 260, "x": 35, "y": 58 } } | 5.79 s |
| banking-ios/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.00 s |
| banking-ios/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.72 s |
| flight-pass-ios/clean/grounding/boarding-pass | 1 | success | Target missed | 4.66 s |
| flight-pass-ios/clean/grounding/check-in | 1 | success | Target hit | 3.46 s |
| flight-pass-ios/clean/grounding/trips-tab | 1 | success | Target hit | 5.15 s |
| flight-pass-ios/clean/grounding/profile-tab | 1 | success | Target hit | 4.18 s |
| flight-pass-ios/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } | 4.41 s |
| flight-pass-ios/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red Flight update banner overlaps blue banner beneath it, obscuring its text.", "has_issue": true, "region": { "height": 62, "width": 912, "x": 43, "y": 863 } } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red Flight update banner overlaps blue banner beneath it, obscuring its text.", "has_issue": true, "region": { "height": 62, "width": 912, "x": 43, "y": 863 } } | 9.53 s |
| flight-pass-ios/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.16 s |
| flight-pass-ios/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.53 s |
| smart-home-android/clean/grounding/more | 1 | success | Target hit | 4.07 s |
| smart-home-android/clean/grounding/room-search | 1 | success | Target hit | 5.98 s |
| smart-home-android/clean/grounding/thermostat-card | 1 | success | Target hit | 4.82 s |
| smart-home-android/clean/grounding/add-device | 1 | success | Target hit | 3.71 s |
| smart-home-android/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } | 4.54 s |
| smart-home-android/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": "viewport_cropping", "description": "Kitchen lights Quick view card is offset right and clipped by viewport edge.", "has_issue": true, "region": { "height": 116, "width": 845, "x": 155, "y": 534 } } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": "viewport_cropping", "description": "Kitchen lights Quick view card is offset right and clipped by viewport edge.", "has_issue": true, "region": { "height": 116, "width": 845, "x": 155, "y": 534 } } | 8.58 s |
| smart-home-android/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.20 s |
| smart-home-android/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 3.74 s |
| commerce-catalog-desktop/clean/grounding/search | 2 | success | Target hit | 4.15 s |
| commerce-catalog-desktop/clean/grounding/add-product | 2 | success | Target hit | 3.72 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 2 | success | Target hit | 2.27 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 2 | success | Target hit | 5.52 s |
| commerce-catalog-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } | 4.06 s |
| commerce-catalog-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Main heading clipped on left showing 'ducts' and 'IANAGEMENT' instead of full 'Products' title.", "has_issue": true, "region": { "height": 68, "width": 69, "x": 191, "y": 125 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Main heading clipped on left showing 'ducts' and 'IANAGEMENT' instead of full 'Products' title.", "has_issue": true, "region": { "height": 68, "width": 69, "x": 191, "y": 125 } } | 10.11 s |
| commerce-catalog-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.76 s |
| commerce-catalog-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 3.51 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 2 | success | Target hit | 4.69 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 2 | success | Target hit | 3.86 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 2 | success | Target hit | 4.04 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 2 | success | Target hit | 7.95 s |
| analytics-dashboard-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } | 4.03 s |
| analytics-dashboard-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card heading truncated at container edge, reading 'chann' instead of full text.", "has_issue": true, "region": { "height": 30, "width": 328, "x": 200, "y": 249 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card heading truncated at container edge, reading 'chann' instead of full text.", "has_issue": true, "region": { "height": 30, "width": 328, "x": 200, "y": 249 } } | 9.23 s |
| analytics-dashboard-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.09 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.88 s |
| iot-control-desktop/clean/grounding/devices-nav | 2 | success | Target hit | 4.10 s |
| iot-control-desktop/clean/grounding/restart-device | 2 | success | Target hit | 3.19 s |
| iot-control-desktop/clean/grounding/auto-mode | 2 | success | Target hit | 10.00 s |
| iot-control-desktop/clean/grounding/pump-card | 2 | success | Target hit | 2.78 s |
| iot-control-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect detected.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect detected.", "has_issue": false, "region": null } | 5.68 s |
| iot-control-desktop/defect/layout-defect | 2 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No visible layout defects detected.", "has_issue": false, "region": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No visible layout defects detected.", "has_issue": false, "region": null } | 8.11 s |
| iot-control-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.31 s |
| iot-control-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.58 s |
| banking-ios/clean/grounding/profile | 2 | success | Target hit | 4.57 s |
| banking-ios/clean/grounding/transaction-search | 2 | success | Target hit | 4.25 s |
| banking-ios/clean/grounding/hide-balance | 2 | success | Target hit | 8.61 s |
| banking-ios/clean/grounding/transfer | 2 | success | Target hit | 3.44 s |
| banking-ios/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed", "has_issue": false, "region": null } | 3.40 s |
| banking-ios/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "The 'Banking' header text is cropped at the left viewport edge, cutting off the first letter.", "has_issue": true, "region": { "height": 55, "width": 250, "x": 30, "y": 80 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "The 'Banking' header text is cropped at the left viewport edge, cutting off the first letter.", "has_issue": true, "region": { "height": 55, "width": 250, "x": 30, "y": 80 } } | 5.50 s |
| banking-ios/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 8.35 s |
| banking-ios/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.99 s |
| flight-pass-ios/clean/grounding/boarding-pass | 2 | success | Target missed | 4.49 s |
| flight-pass-ios/clean/grounding/check-in | 2 | success | Target hit | 4.15 s |
| flight-pass-ios/clean/grounding/trips-tab | 2 | success | Target hit | 6.30 s |
| flight-pass-ios/clean/grounding/profile-tab | 2 | success | Target hit | 5.09 s |
| flight-pass-ios/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } | 4.56 s |
| flight-pass-ios/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red Flight update banner overlaps blue banner behind it, obscuring its text.", "has_issue": true, "region": { "height": 55, "width": 910, "x": 45, "y": 875 } } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red Flight update banner overlaps blue banner behind it, obscuring its text.", "has_issue": true, "region": { "height": 55, "width": 910, "x": 45, "y": 875 } } | 10.19 s |
| flight-pass-ios/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.61 s |
| flight-pass-ios/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.10 s |
| smart-home-android/clean/grounding/more | 2 | success | Target hit | 6.58 s |
| smart-home-android/clean/grounding/room-search | 2 | success | Target hit | 4.87 s |
| smart-home-android/clean/grounding/thermostat-card | 2 | success | Target hit | 4.45 s |
| smart-home-android/clean/grounding/add-device | 2 | success | Target hit | 4.54 s |
| smart-home-android/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect.", "has_issue": false, "region": null } | 4.85 s |
| smart-home-android/defect/layout-defect | 2 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": null, "description": "No layout defect detected", "has_issue": false, "region": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": null, "description": "No layout defect detected", "has_issue": false, "region": null } | 9.62 s |
| smart-home-android/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.87 s |
| smart-home-android/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.64 s |
| commerce-catalog-desktop/clean/grounding/search | 3 | success | Target hit | 4.92 s |
| commerce-catalog-desktop/clean/grounding/add-product | 3 | success | Target hit | 3.59 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 3 | success | Target hit | 3.02 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 3 | success | Target hit | 5.15 s |
| commerce-catalog-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect.", "has_issue": false, "region": null } | 4.52 s |
| commerce-catalog-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Page heading clipped on left edge showing 'ducts' instead of 'Products' with kicker text cut off.", "has_issue": true, "region": { "height": 68, "width": 69, "x": 190, "y": 124 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Page heading clipped on left edge showing 'ducts' instead of 'Products' with kicker text cut off.", "has_issue": true, "region": { "height": 68, "width": 69, "x": 190, "y": 124 } } | 5.27 s |
| commerce-catalog-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.90 s |
| commerce-catalog-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 4.55 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 3 | success | Target hit | 5.64 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 3 | success | Target hit | 3.06 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 3 | success | Target hit | 3.71 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 3 | success | Target hit | 5.52 s |
| analytics-dashboard-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect detected.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect detected.", "has_issue": false, "region": null } | 3.62 s |
| analytics-dashboard-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card heading truncated to 'Monthly revenue across all sales chann', clipped at container edge.", "has_issue": true, "region": { "height": 30, "width": 330, "x": 200, "y": 250 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card heading truncated to 'Monthly revenue across all sales chann', clipped at container edge.", "has_issue": true, "region": { "height": 30, "width": 330, "x": 200, "y": 250 } } | 8.87 s |
| analytics-dashboard-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.07 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.90 s |
| iot-control-desktop/clean/grounding/devices-nav | 3 | success | Target hit | 3.49 s |
| iot-control-desktop/clean/grounding/restart-device | 3 | success | Target hit | 4.07 s |
| iot-control-desktop/clean/grounding/auto-mode | 3 | success | Target hit | 5.22 s |
| iot-control-desktop/clean/grounding/pump-card | 3 | success | Target hit | 5.83 s |
| iot-control-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect detected.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect detected.", "has_issue": false, "region": null } | 4.75 s |
| iot-control-desktop/defect/layout-defect | 3 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No layout defect detected", "has_issue": false, "region": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No layout defect detected", "has_issue": false, "region": null } | 7.46 s |
| iot-control-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 4.12 s |
| iot-control-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.87 s |
| banking-ios/clean/grounding/profile | 3 | success | Target hit | 4.00 s |
| banking-ios/clean/grounding/transaction-search | 3 | success | Target hit | 5.64 s |
| banking-ios/clean/grounding/hide-balance | 3 | success | Target hit | 6.21 s |
| banking-ios/clean/grounding/transfer | 3 | success | Target hit | 3.53 s |
| banking-ios/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } | 5.26 s |
| banking-ios/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "Header title Banking and top brand text truncated at left viewport edge with B cropped.", "has_issue": true, "region": { "height": 75, "width": 270, "x": 10, "y": 35 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "Header title Banking and top brand text truncated at left viewport edge with B cropped.", "has_issue": true, "region": { "height": 75, "width": 270, "x": 10, "y": 35 } } | 7.26 s |
| banking-ios/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.34 s |
| banking-ios/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 3.49 s |
| flight-pass-ios/clean/grounding/boarding-pass | 3 | success | Target missed | 3.76 s |
| flight-pass-ios/clean/grounding/check-in | 3 | success | Target hit | 4.72 s |
| flight-pass-ios/clean/grounding/trips-tab | 3 | success | Target hit | 4.97 s |
| flight-pass-ios/clean/grounding/profile-tab | 3 | success | Target hit | 5.65 s |
| flight-pass-ios/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect detected.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect detected.", "has_issue": false, "region": null } | 5.01 s |
| flight-pass-ios/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red flight update banner overlaps blue banner behind it, obscuring its text", "has_issue": true, "region": { "height": 75, "width": 920, "x": 40, "y": 840 } } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red flight update banner overlaps blue banner behind it, obscuring its text", "has_issue": true, "region": { "height": 75, "width": 920, "x": 40, "y": 840 } } | 5.54 s |
| flight-pass-ios/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.05 s |
| flight-pass-ios/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.99 s |
| smart-home-android/clean/grounding/more | 3 | success | Target hit | 10.73 s |
| smart-home-android/clean/grounding/room-search | 3 | success | Target hit | 4.38 s |
| smart-home-android/clean/grounding/thermostat-card | 3 | success | Target hit | 4.20 s |
| smart-home-android/clean/grounding/add-device | 3 | success | Target hit | 3.82 s |
| smart-home-android/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defects observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defects observed.", "has_issue": false, "region": null } | 4.26 s |
| smart-home-android/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": "broken_responsive_reflow", "description": "The Kitchen lights card is indented and narrower than other cards, breaking list alignment.", "has_issue": true, "region": { "height": 90, "width": 675, "x": 280, "y": 530 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": "broken_responsive_reflow", "description": "The Kitchen lights card is indented and narrower than other cards, breaking list alignment.", "has_issue": true, "region": { "height": 90, "width": 675, "x": 280, "y": 530 } } | 11.76 s |
| smart-home-android/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.94 s |
| smart-home-android/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.33 s |
GPT-5.6 Luna · 144 case executions
Returned model versions: openai/gpt-5.6-luna.
Targets with the same hit/miss outcome across all three repetitions: 23 / 24. Agreement can include consistently missing a target.
| Case | Repeat | Result | Outcome | Time |
|---|---|---|---|---|
| commerce-catalog-desktop/clean/grounding/search | 1 | success | Target hit | 2.56 s |
| commerce-catalog-desktop/clean/grounding/add-product | 1 | success | Target hit | 2.00 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 1 | success | Target hit | 2.45 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 1 | success | Target hit | 2.89 s |
| commerce-catalog-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.03 s |
| commerce-catalog-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 127, "width": 70, "height": 62 }, "description": "The left portions of the page heading text are visibly clipped, showing “IAGEMENT” and “ducts” instead of the full labels." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 127, "width": 70, "height": 62 }, "description": "The left portions of the page heading text are visibly clipped, showing “IAGEMENT” and “ducts” instead of the full labels." } | 4.66 s |
| commerce-catalog-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.85 s |
| commerce-catalog-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.73 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 1 | success | Target hit | 5.28 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 1 | success | Target hit | 2.12 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 1 | success | Target hit | 1.87 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 1 | success | Target hit | 3.44 s |
| analytics-dashboard-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.97 s |
| analytics-dashboard-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 250, "width": 326, "height": 34 }, "description": "The first summary card's heading is visibly truncated at the card's right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 250, "width": 326, "height": 34 }, "description": "The first summary card's heading is visibly truncated at the card's right edge." } | 3.08 s |
| analytics-dashboard-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.31 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.29 s |
| iot-control-desktop/clean/grounding/devices-nav | 1 | success | Target hit | 2.47 s |
| iot-control-desktop/clean/grounding/restart-device | 1 | success | Target hit | 2.03 s |
| iot-control-desktop/clean/grounding/auto-mode | 1 | success | Target hit | 4.02 s |
| iot-control-desktop/clean/grounding/pump-card | 1 | success | Target hit | 2.64 s |
| iot-control-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.91 s |
| iot-control-desktop/defect/layout-defect | 1 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 3.79 s |
| iot-control-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.55 s |
| iot-control-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.29 s |
| banking-ios/clean/grounding/profile | 1 | success | Target hit | 2.82 s |
| banking-ios/clean/grounding/transaction-search | 1 | success | Target missed | 1.87 s |
| banking-ios/clean/grounding/hide-balance | 1 | success | Target hit | 3.78 s |
| banking-ios/clean/grounding/transfer | 1 | success | Target hit | 1.69 s |
| banking-ios/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.95 s |
| banking-ios/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 44, "y": 71, "width": 244, "height": 33 }, "description": "The left edge of the large “Banking” heading is visibly cropped, leaving the initial character partially cut off." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 44, "y": 71, "width": 244, "height": 33 }, "description": "The left edge of the large “Banking” heading is visibly cropped, leaving the initial character partially cut off." } | 6.58 s |
| banking-ios/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.64 s |
| banking-ios/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.33 s |
| flight-pass-ios/clean/grounding/boarding-pass | 1 | success | Target hit | 2.10 s |
| flight-pass-ios/clean/grounding/check-in | 1 | success | Target hit | 2.62 s |
| flight-pass-ios/clean/grounding/trips-tab | 1 | success | Target hit | 4.46 s |
| flight-pass-ios/clean/grounding/profile-tab | 1 | success | Target hit | 5.20 s |
| flight-pass-ios/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.61 s |
| flight-pass-ios/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "A red flight update banner visibly covers the lower portion of a blue actionable button near the bottom of the screen." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "A red flight update banner visibly covers the lower portion of a blue actionable button near the bottom of the screen." } | 3.27 s |
| flight-pass-ios/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.40 s |
| flight-pass-ios/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.14 s |
| smart-home-android/clean/grounding/more | 1 | success | Target hit | 2.51 s |
| smart-home-android/clean/grounding/room-search | 1 | success | Target hit | 1.95 s |
| smart-home-android/clean/grounding/thermostat-card | 1 | success | Target hit | 2.74 s |
| smart-home-android/clean/grounding/add-device | 1 | success | Target hit | 2.44 s |
| smart-home-android/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.88 s |
| smart-home-android/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 510, "width": 600, "height": 69 }, "description": "The second Quick view card is visibly offset to the right and does not align with the other cards in the list." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 510, "width": 600, "height": 69 }, "description": "The second Quick view card is visibly offset to the right and does not align with the other cards in the list." } | 5.45 s |
| smart-home-android/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.57 s |
| smart-home-android/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.57 s |
| commerce-catalog-desktop/clean/grounding/search | 2 | success | Target hit | 2.62 s |
| commerce-catalog-desktop/clean/grounding/add-product | 2 | success | Target hit | 2.64 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 2 | success | Target hit | 2.37 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 2 | success | Target hit | 3.21 s |
| commerce-catalog-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.91 s |
| commerce-catalog-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 192, "y": 151, "width": 72, "height": 39 }, "description": "The page title is visibly clipped on the left, displaying only “ducts” instead of the full title." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 192, "y": 151, "width": 72, "height": 39 }, "description": "The page title is visibly clipped on the left, displaying only “ducts” instead of the full title." } | 4.33 s |
| commerce-catalog-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.17 s |
| commerce-catalog-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.70 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 2 | success | Target hit | 4.50 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 2 | success | Target hit | 2.67 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 2 | success | Target hit | 2.36 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 2 | success | Target hit | 3.04 s |
| analytics-dashboard-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.84 s |
| analytics-dashboard-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 252, "width": 325, "height": 28 }, "description": "The first summary card's headline is visibly truncated at the right edge, ending mid-word." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 252, "width": 325, "height": 28 }, "description": "The first summary card's headline is visibly truncated at the right edge, ending mid-word." } | 2.88 s |
| analytics-dashboard-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.57 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.66 s |
| iot-control-desktop/clean/grounding/devices-nav | 2 | success | Target hit | 3.05 s |
| iot-control-desktop/clean/grounding/restart-device | 2 | success | Target hit | 2.25 s |
| iot-control-desktop/clean/grounding/auto-mode | 2 | success | Target hit | 2.80 s |
| iot-control-desktop/clean/grounding/pump-card | 2 | success | Target hit | 3.40 s |
| iot-control-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.31 s |
| iot-control-desktop/defect/layout-defect | 2 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.85 s |
| iot-control-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.08 s |
| iot-control-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.07 s |
| banking-ios/clean/grounding/profile | 2 | success | Target hit | 3.15 s |
| banking-ios/clean/grounding/transaction-search | 2 | success | Target hit | 2.17 s |
| banking-ios/clean/grounding/hide-balance | 2 | success | Target hit | 3.60 s |
| banking-ios/clean/grounding/transfer | 2 | success | Target hit | 2.21 s |
| banking-ios/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.69 s |
| banking-ios/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 45, "y": 70, "width": 245, "height": 32 }, "description": "The left edge of the “Banking” header text is visibly clipped, with the initial letter cut off." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 45, "y": 70, "width": 245, "height": 32 }, "description": "The left edge of the “Banking” header text is visibly clipped, with the initial letter cut off." } | 4.84 s |
| banking-ios/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.59 s |
| banking-ios/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.55 s |
| flight-pass-ios/clean/grounding/boarding-pass | 2 | success | Target hit | 3.22 s |
| flight-pass-ios/clean/grounding/check-in | 2 | success | Target hit | 2.06 s |
| flight-pass-ios/clean/grounding/trips-tab | 2 | success | Target hit | 7.16 s |
| flight-pass-ios/clean/grounding/profile-tab | 2 | success | Target hit | 3.82 s |
| flight-pass-ios/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.55 s |
| flight-pass-ios/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 64 }, "description": "A red flight-update banner visibly overlaps and obscures the blue actionable button near the bottom of the screen." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 64 }, "description": "A red flight-update banner visibly overlaps and obscures the blue actionable button near the bottom of the screen." } | 3.37 s |
| flight-pass-ios/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.73 s |
| flight-pass-ios/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.64 s |
| smart-home-android/clean/grounding/more | 2 | success | Target hit | 2.67 s |
| smart-home-android/clean/grounding/room-search | 2 | success | Target hit | 2.30 s |
| smart-home-android/clean/grounding/thermostat-card | 2 | success | Target hit | 2.26 s |
| smart-home-android/clean/grounding/add-device | 2 | success | Target hit | 1.93 s |
| smart-home-android/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.57 s |
| smart-home-android/defect/layout-defect | 2 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.87 s |
| smart-home-android/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.17 s |
| smart-home-android/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.36 s |
| commerce-catalog-desktop/clean/grounding/search | 3 | success | Target hit | 2.61 s |
| commerce-catalog-desktop/clean/grounding/add-product | 3 | success | Target hit | 2.88 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 3 | success | Target hit | 4.31 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 3 | success | Target hit | 3.36 s |
| commerce-catalog-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.36 s |
| commerce-catalog-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 76, "height": 64 }, "description": "The page title is visibly clipped on the left, displaying “ducts” instead of the full “Products”." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 76, "height": 64 }, "description": "The page title is visibly clipped on the left, displaying “ducts” instead of the full “Products”." } | 3.66 s |
| commerce-catalog-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.69 s |
| commerce-catalog-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.80 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 3 | success | Target hit | 3.61 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 3 | success | Target hit | 2.33 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 3 | success | Target hit | 2.09 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 3 | success | Target hit | 3.14 s |
| analytics-dashboard-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.33 s |
| analytics-dashboard-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 253, "width": 326, "height": 28 }, "description": "The first summary card's heading is visibly cut off at the right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 253, "width": 326, "height": 28 }, "description": "The first summary card's heading is visibly cut off at the right edge." } | 3.52 s |
| analytics-dashboard-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.61 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.64 s |
| iot-control-desktop/clean/grounding/devices-nav | 3 | success | Target hit | 2.44 s |
| iot-control-desktop/clean/grounding/restart-device | 3 | success | Target hit | 3.00 s |
| iot-control-desktop/clean/grounding/auto-mode | 3 | success | Target hit | 3.80 s |
| iot-control-desktop/clean/grounding/pump-card | 3 | success | Target hit | 3.33 s |
| iot-control-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.57 s |
| iot-control-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 463, "y": 558, "width": 222, "height": 66 }, "description": "A maintenance notification overlays the device table content." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 463, "y": 558, "width": 222, "height": 66 }, "description": "A maintenance notification overlays the device table content." } | 3.65 s |
| iot-control-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.82 s |
| iot-control-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.48 s |
| banking-ios/clean/grounding/profile | 3 | success | Target hit | 2.94 s |
| banking-ios/clean/grounding/transaction-search | 3 | success | Target hit | 2.50 s |
| banking-ios/clean/grounding/hide-balance | 3 | success | Target hit | 4.97 s |
| banking-ios/clean/grounding/transfer | 3 | success | Target hit | 2.55 s |
| banking-ios/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.96 s |
| banking-ios/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 44, "y": 72, "width": 244, "height": 29 }, "description": "The left edge of the Banking heading is visibly clipped, cutting off the first character." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 44, "y": 72, "width": 244, "height": 29 }, "description": "The left edge of the Banking heading is visibly clipped, cutting off the first character." } | 3.82 s |
| banking-ios/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.26 s |
| banking-ios/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.30 s |
| flight-pass-ios/clean/grounding/boarding-pass | 3 | success | Target hit | 2.52 s |
| flight-pass-ios/clean/grounding/check-in | 3 | success | Target hit | 2.24 s |
| flight-pass-ios/clean/grounding/trips-tab | 3 | success | Target hit | 4.28 s |
| flight-pass-ios/clean/grounding/profile-tab | 3 | success | Target hit | 3.22 s |
| flight-pass-ios/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.70 s |
| flight-pass-ios/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 63 }, "description": "A red flight update banner obscures the lower portion of a blue actionable button near the bottom of the screen." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 63 }, "description": "A red flight update banner obscures the lower portion of a blue actionable button near the bottom of the screen." } | 3.14 s |
| flight-pass-ios/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.40 s |
| flight-pass-ios/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.32 s |
| smart-home-android/clean/grounding/more | 3 | success | Target hit | 2.35 s |
| smart-home-android/clean/grounding/room-search | 3 | success | Target hit | 2.65 s |
| smart-home-android/clean/grounding/thermostat-card | 3 | success | Target hit | 2.64 s |
| smart-home-android/clean/grounding/add-device | 3 | success | Target hit | 1.92 s |
| smart-home-android/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.88 s |
| smart-home-android/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 510, "width": 600, "height": 69 }, "description": "The second Quick view card is misaligned and compressed to the right compared with the other full-width cards." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 510, "width": 600, "height": 69 }, "description": "The second Quick view card is misaligned and compressed to the right compared with the other full-width cards." } | 4.82 s |
| smart-home-android/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.19 s |
| smart-home-android/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.62 s |
Jev 1.14 · 144 case executions
Returned model versions: typesafe/jev-1.14-20261014.
Targets with the same hit/miss outcome across all three repetitions: 24 / 24. Agreement can include consistently missing a target.
| Case | Repeat | Result | Outcome | Time |
|---|---|---|---|---|
| commerce-catalog-desktop/clean/grounding/search | 1 | success | Target hit | 0.43 s |
| commerce-catalog-desktop/clean/grounding/add-product | 1 | success | Target hit | 0.41 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 1 | success | Target hit | 0.40 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 1 | success | Target hit | 0.38 s |
| commerce-catalog-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.47 s |
| commerce-catalog-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 190, "y": 140, "width": 60, "height": 69.99999999999999 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 190, "y": 140, "width": 60, "height": 69.99999999999999 }, "description": "" } | 0.39 s |
| commerce-catalog-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.35 s |
| commerce-catalog-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.44 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 1 | success | Target hit | 0.53 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 1 | success | Target hit | 0.37 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 1 | success | Target hit | 0.38 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 1 | success | Target hit | 0.34 s |
| analytics-dashboard-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.48 s |
| analytics-dashboard-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 360.00000000000006, "height": 110.00000000000001 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 360.00000000000006, "height": 110.00000000000001 }, "description": "" } | 0.36 s |
| analytics-dashboard-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.33 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.39 s |
| iot-control-desktop/clean/grounding/devices-nav | 1 | success | Target hit | 0.34 s |
| iot-control-desktop/clean/grounding/restart-device | 1 | success | Target hit | 0.32 s |
| iot-control-desktop/clean/grounding/auto-mode | 1 | success | Target hit | 0.41 s |
| iot-control-desktop/clean/grounding/pump-card | 1 | success | Target hit | 0.37 s |
| iot-control-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.48 s |
| iot-control-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 450, "y": 550, "width": 239.99999999999994, "height": 89.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 450, "y": 550, "width": 239.99999999999994, "height": 89.99999999999997 }, "description": "" } | 0.38 s |
| iot-control-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.35 s |
| iot-control-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.46 s |
| banking-ios/clean/grounding/profile | 1 | success | Target hit | 0.39 s |
| banking-ios/clean/grounding/transaction-search | 1 | success | Target hit | 0.48 s |
| banking-ios/clean/grounding/hide-balance | 1 | success | Target hit | 0.41 s |
| banking-ios/clean/grounding/transfer | 1 | success | Target hit | 0.43 s |
| banking-ios/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.37 s |
| banking-ios/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" } | 0.30 s |
| banking-ios/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.37 s |
| banking-ios/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.38 s |
| flight-pass-ios/clean/grounding/boarding-pass | 1 | success | Target hit | 0.31 s |
| flight-pass-ios/clean/grounding/check-in | 1 | success | Target hit | 0.48 s |
| flight-pass-ios/clean/grounding/trips-tab | 1 | success | Target hit | 0.34 s |
| flight-pass-ios/clean/grounding/profile-tab | 1 | success | Target hit | 0.32 s |
| flight-pass-ios/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.30 s |
| flight-pass-ios/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 70.00000000000006 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 70.00000000000006 }, "description": "" } | 0.37 s |
| flight-pass-ios/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.28 s |
| flight-pass-ios/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.38 s |
| smart-home-android/clean/grounding/more | 1 | success | Target hit | 0.36 s |
| smart-home-android/clean/grounding/room-search | 1 | success | Target hit | 0.34 s |
| smart-home-android/clean/grounding/thermostat-card | 1 | success | Target hit | 0.37 s |
| smart-home-android/clean/grounding/add-device | 1 | success | Target hit | 0.33 s |
| smart-home-android/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.43 s |
| smart-home-android/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 470, "width": 590, "height": 99.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 470, "width": 590, "height": 99.99999999999997 }, "description": "" } | 0.39 s |
| smart-home-android/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.38 s |
| smart-home-android/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.41 s |
| commerce-catalog-desktop/clean/grounding/search | 2 | success | Target hit | 0.44 s |
| commerce-catalog-desktop/clean/grounding/add-product | 2 | success | Target hit | 0.42 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 2 | success | Target hit | 0.38 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 2 | success | Target hit | 0.42 s |
| commerce-catalog-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.38 s |
| commerce-catalog-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 180, "y": 140, "width": 70, "height": 69.99999999999999 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 180, "y": 140, "width": 70, "height": 69.99999999999999 }, "description": "" } | 0.39 s |
| commerce-catalog-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.38 s |
| commerce-catalog-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.54 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 2 | success | Target hit | 0.41 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 2 | success | Target hit | 0.66 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 2 | success | Target hit | 0.48 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 2 | success | Target hit | 0.37 s |
| analytics-dashboard-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.39 s |
| analytics-dashboard-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 360.00000000000006, "height": 100 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 360.00000000000006, "height": 100 }, "description": "" } | 0.39 s |
| analytics-dashboard-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.36 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.87 s |
| iot-control-desktop/clean/grounding/devices-nav | 2 | success | Target hit | 0.46 s |
| iot-control-desktop/clean/grounding/restart-device | 2 | success | Target hit | 0.66 s |
| iot-control-desktop/clean/grounding/auto-mode | 2 | success | Target hit | 0.39 s |
| iot-control-desktop/clean/grounding/pump-card | 2 | success | Target hit | 0.35 s |
| iot-control-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.33 s |
| iot-control-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 460, "y": 540, "width": 239.99999999999994, "height": 99.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 460, "y": 540, "width": 239.99999999999994, "height": 99.99999999999997 }, "description": "" } | 0.37 s |
| iot-control-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.33 s |
| iot-control-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.38 s |
| banking-ios/clean/grounding/profile | 2 | success | Target hit | 0.33 s |
| banking-ios/clean/grounding/transaction-search | 2 | success | Target hit | 0.30 s |
| banking-ios/clean/grounding/hide-balance | 2 | success | Target hit | 0.33 s |
| banking-ios/clean/grounding/transfer | 2 | success | Target hit | 0.41 s |
| banking-ios/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.30 s |
| banking-ios/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" } | 0.44 s |
| banking-ios/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.31 s |
| banking-ios/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.29 s |
| flight-pass-ios/clean/grounding/boarding-pass | 2 | success | Target hit | 0.30 s |
| flight-pass-ios/clean/grounding/check-in | 2 | success | Target hit | 0.29 s |
| flight-pass-ios/clean/grounding/trips-tab | 2 | success | Target hit | 0.27 s |
| flight-pass-ios/clean/grounding/profile-tab | 2 | success | Target hit | 0.31 s |
| flight-pass-ios/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.32 s |
| flight-pass-ios/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 70.00000000000006 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 70.00000000000006 }, "description": "" } | 0.31 s |
| flight-pass-ios/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.29 s |
| flight-pass-ios/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.29 s |
| smart-home-android/clean/grounding/more | 2 | success | Target hit | 0.32 s |
| smart-home-android/clean/grounding/room-search | 2 | success | Target hit | 0.30 s |
| smart-home-android/clean/grounding/thermostat-card | 2 | success | Target hit | 0.38 s |
| smart-home-android/clean/grounding/add-device | 2 | success | Target hit | 0.32 s |
| smart-home-android/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.29 s |
| smart-home-android/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 480, "width": 590, "height": 99.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 480, "width": 590, "height": 99.99999999999997 }, "description": "" } | 0.30 s |
| smart-home-android/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.32 s |
| smart-home-android/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.35 s |
| commerce-catalog-desktop/clean/grounding/search | 3 | success | Target hit | 0.35 s |
| commerce-catalog-desktop/clean/grounding/add-product | 3 | success | Target hit | 0.37 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 3 | success | Target hit | 0.40 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 3 | success | Target hit | 0.45 s |
| commerce-catalog-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.37 s |
| commerce-catalog-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 180, "y": 140, "width": 70, "height": 69.99999999999999 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 180, "y": 140, "width": 70, "height": 69.99999999999999 }, "description": "" } | 0.38 s |
| commerce-catalog-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.39 s |
| commerce-catalog-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.43 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 3 | success | Target hit | 0.33 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 3 | success | Target hit | 0.35 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 3 | success | Target hit | 0.36 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 3 | success | Target hit | 0.38 s |
| analytics-dashboard-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.35 s |
| analytics-dashboard-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 350.00000000000006, "height": 110.00000000000001 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 350.00000000000006, "height": 110.00000000000001 }, "description": "" } | 0.36 s |
| analytics-dashboard-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.36 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.39 s |
| iot-control-desktop/clean/grounding/devices-nav | 3 | success | Target hit | 0.33 s |
| iot-control-desktop/clean/grounding/restart-device | 3 | success | Target hit | 0.32 s |
| iot-control-desktop/clean/grounding/auto-mode | 3 | success | Target hit | 0.34 s |
| iot-control-desktop/clean/grounding/pump-card | 3 | success | Target hit | 0.35 s |
| iot-control-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.37 s |
| iot-control-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 460, "y": 550, "width": 229.99999999999991, "height": 89.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 460, "y": 550, "width": 229.99999999999991, "height": 89.99999999999997 }, "description": "" } | 0.37 s |
| iot-control-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.36 s |
| iot-control-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.44 s |
| banking-ios/clean/grounding/profile | 3 | success | Target hit | 0.29 s |
| banking-ios/clean/grounding/transaction-search | 3 | success | Target hit | 0.29 s |
| banking-ios/clean/grounding/hide-balance | 3 | success | Target hit | 0.29 s |
| banking-ios/clean/grounding/transfer | 3 | success | Target hit | 0.33 s |
| banking-ios/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.44 s |
| banking-ios/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" } | 0.30 s |
| banking-ios/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.28 s |
| banking-ios/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.44 s |
| flight-pass-ios/clean/grounding/boarding-pass | 3 | success | Target hit | 0.32 s |
| flight-pass-ios/clean/grounding/check-in | 3 | success | Target hit | 0.30 s |
| flight-pass-ios/clean/grounding/trips-tab | 3 | success | Target hit | 0.29 s |
| flight-pass-ios/clean/grounding/profile-tab | 3 | success | Target hit | 0.28 s |
| flight-pass-ios/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.36 s |
| flight-pass-ios/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 60.00000000000006 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 60.00000000000006 }, "description": "" } | 0.30 s |
| flight-pass-ios/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.29 s |
| flight-pass-ios/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.29 s |
| smart-home-android/clean/grounding/more | 3 | success | Target hit | 0.29 s |
| smart-home-android/clean/grounding/room-search | 3 | success | Target hit | 0.39 s |
| smart-home-android/clean/grounding/thermostat-card | 3 | success | Target hit | 0.35 s |
| smart-home-android/clean/grounding/add-device | 3 | success | Target hit | 0.35 s |
| smart-home-android/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.35 s |
| smart-home-android/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 480, "width": 590, "height": 99.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 480, "width": 590, "height": 99.99999999999997 }, "description": "" } | 0.53 s |
| smart-home-android/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.39 s |
| smart-home-android/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.34 s |
GPT-6 Luna Decisions · 72 case executions
Returned model versions: openai/gpt-6-luna-decisions-20261006.
| Case | Repeat | Result | Outcome | Time |
|---|---|---|---|---|
| commerce-catalog-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.38 s |
| commerce-catalog-desktop/defect/layout-defect | 1 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null } | 0.45 s |
| commerce-catalog-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.32 s |
| commerce-catalog-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.35 s |
| analytics-dashboard-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.38 s |
| analytics-dashboard-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" } | 0.36 s |
| analytics-dashboard-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.30 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.38 s |
| iot-control-desktop/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.32 s |
| iot-control-desktop/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" } | 0.52 s |
| iot-control-desktop/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.55 s |
| iot-control-desktop/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.26 s |
| banking-ios/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.24 s |
| banking-ios/defect/layout-defect | 1 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null } | 0.21 s |
| banking-ios/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.29 s |
| banking-ios/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.22 s |
| flight-pass-ios/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.53 s |
| flight-pass-ios/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" } | 0.66 s |
| flight-pass-ios/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.38 s |
| flight-pass-ios/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.19 s |
| smart-home-android/clean/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.33 s |
| smart-home-android/defect/layout-defect | 1 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" } | 0.27 s |
| smart-home-android/clean/element-presence/present | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.26 s |
| smart-home-android/clean/element-presence/absent | 1 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.21 s |
| commerce-catalog-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.35 s |
| commerce-catalog-desktop/defect/layout-defect | 2 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null } | 0.38 s |
| commerce-catalog-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.27 s |
| commerce-catalog-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.36 s |
| analytics-dashboard-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.35 s |
| analytics-dashboard-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" } | 0.32 s |
| analytics-dashboard-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.47 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.28 s |
| iot-control-desktop/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.74 s |
| iot-control-desktop/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" } | 0.60 s |
| iot-control-desktop/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.91 s |
| iot-control-desktop/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.41 s |
| banking-ios/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.24 s |
| banking-ios/defect/layout-defect | 2 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null } | 0.21 s |
| banking-ios/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.49 s |
| banking-ios/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.24 s |
| flight-pass-ios/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.62 s |
| flight-pass-ios/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" } | 0.27 s |
| flight-pass-ios/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.26 s |
| flight-pass-ios/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.23 s |
| smart-home-android/clean/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.24 s |
| smart-home-android/defect/layout-defect | 2 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" } | 0.53 s |
| smart-home-android/clean/element-presence/present | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.29 s |
| smart-home-android/clean/element-presence/absent | 2 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.19 s |
| commerce-catalog-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.49 s |
| commerce-catalog-desktop/defect/layout-defect | 3 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null } | 0.25 s |
| commerce-catalog-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.50 s |
| commerce-catalog-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.48 s |
| analytics-dashboard-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.36 s |
| analytics-dashboard-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" } | 0.23 s |
| analytics-dashboard-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.32 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.52 s |
| iot-control-desktop/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.36 s |
| iot-control-desktop/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" } | 0.37 s |
| iot-control-desktop/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.31 s |
| iot-control-desktop/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.28 s |
| banking-ios/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.25 s |
| banking-ios/defect/layout-defect | 3 | success | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null } | 0.25 s |
| banking-ios/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.27 s |
| banking-ios/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.52 s |
| flight-pass-ios/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.54 s |
| flight-pass-ios/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" } | 1.22 s |
| flight-pass-ios/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.19 s |
| flight-pass-ios/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.89 s |
| smart-home-android/clean/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 1.09 s |
| smart-home-android/defect/layout-defect | 3 | success | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" } | 0.25 s |
| smart-home-android/clean/element-presence/present | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.21 s |
| smart-home-android/clean/element-presence/absent | 3 | success | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.32 s |
The completed pilot
Recorded case executions: 1368 / 1368. HTTP attempts: 1368. Reported API cost: $3.3802.
Nine configurations can return target coordinates. GPT-6 Luna Decisions answers classification questions and is excluded from target-localization results.
Methodology
Measure what the model actually sees
The experiment sections document their own scoring rules. This shared methodology covers the common input contract, provider operations, research context, and limits.
InputScreenshot only
Geometry0–1000
RunsThree per case
ScoringSeparate metrics
Operations
Response time, failures, and cost
HTTP round trip and local parsing for final retained attempts; no generation-metadata polling. All transport attempts contribute to reported cost. The classification-only configuration runs half as many cases.
| Model | Completed / planned | Final request failures | Invalid responses | Median | P95 | Reported cost | Cost per 1,000 requests |
|---|---|---|---|---|---|---|---|
| GPT-6 Luna | 144 / 144 | 0 | 0 | 2.18 s | 3.83 s | $0.0187 | $0.1296 |
| GPT-6.1 Sol | 144 / 144 | 0 | 0 | 2.68 s | 4.88 s | $0.2803 | $1.9468 |
| Claude Sonnet 5.5 | 144 / 144 | 0 | 8 | 2.20 s | 3.99 s | $0.5803 | $4.0297 |
| Gemini 3.8 Flash | 144 / 144 | 0 | 1 | 2.42 s | 10.96 s | $0.2655 | $1.8437 |
| Grok 4.7 | 144 / 144 | 0 | 2 | 12.03 s | 28.38 s | $1.1171 | $7.7580 |
| Qwen3.8 Max (0902) | 144 / 144 | 0 | 1 | 8.14 s | 14.76 s | $0.6158 | $4.2762 |
| Muse Spark 1.3 | 144 / 144 | 0 | 0 | 4.22 s | 9.18 s | $0.4342 | $3.0151 |
| GPT-5.6 Luna | 144 / 144 | 0 | 0 | 2.34 s | 4.80 s | $0.0440 | $0.3057 |
| Jev 1.14 | 144 / 144 | 0 | 0 | 0.37 s | 0.48 s | $0.0159 | $0.1104 |
| GPT-6 Luna Decisions | 72 / 72 | 0 | 0 | 0.34 s | 0.90 s | $0.0085 | $0.1177 |
Presence checks: the shared yes/no task
Compare response time and cost on the same presence cases. Layout requests from general models also require a region, so their timing is not the same workload as classification-only Decisions.
| Model | Presence cases | Median | P95 | Presence cost per 1,000 requests |
|---|---|---|---|---|
| GPT-6 Luna | 36 | 1.45 s | 1.88 s | $0.0752 |
| GPT-6.1 Sol | 36 | 2.18 s | 2.68 s | $1.4487 |
| Claude Sonnet 5.5 | 36 | 1.85 s | 2.78 s | $2.9955 |
| Gemini 3.8 Flash | 36 | 2.45 s | 2.98 s | $1.2008 |
| Grok 4.7 | 36 | 1.97 s | 5.95 s | $2.8797 |
| Qwen3.8 Max (0902) | 36 | 4.51 s | 6.79 s | $2.6708 |
| Muse Spark 1.3 | 36 | 2.06 s | 4.23 s | $2.0156 |
| GPT-5.6 Luna | 36 | 1.58 s | 2.11 s | $0.1533 |
| Jev 1.14 | 36 | 0.37 s | 0.48 s | $0.1069 |
| GPT-6 Luna Decisions | 36 | 0.31 s | 0.89 s | $0.1062 |
Methodology
What changed, and what stayed frozen
- The six screenshots, annotations, task prompts, and original response schemas come from the 2026-08-13 corpus. The new edition retains its own copies.
- Each case runs three times. Generative models request low reasoning effort and a 2,048-token output allowance, increased from the original 400-token setting. Providers can report additional billed reasoning tokens above that allowance. This is a disclosed execution-contract change.
- Requests are pinned to healthy standard first-party endpoints with fallbacks disabled. Exact endpoints, prices, settings, and hashes are retained.
- Decision models receive native questions alongside the task. Noul probabilities use a fixed 0.5 threshold. Jev's click point is computed from its box centre; other geometry-capable models predict a separate click point.
- Luna Decisions supports classification, not Box output. Its layout contract therefore measures defect presence and category only. Native typed output and adapter-generated JSON validity are not equivalent to free JSON generation.
- Invalid model answers are not repaired or rerun. Only transient transport failures can receive one retry; every attempt is retained. Random provider and network variation can affect response times. Reported costs include any automatic cache discounts; later repetitions may reuse warm caches.
Three repetitions measure consistency on these cases. Six synthetic screens cannot establish broad model superiority, production reliability, or multi-step navigation capability. No device action, recovery, or navigation-goal completion is tested.
Input contract
Screenshot only
Prompt text plus one base64 PNG—no DOM, accessibility tree, source, OCR, tools, plugins, or web search.
Fixed rendering
Chromium at 1440 × 900 or 390 × 844; device scale factor 1.
Normalized geometry
Browser-derived ground truth in a 0–1000 coordinate space.
Stable pairs
Clean and defective variants preserve target geometry and capture hashes.
Operations
Keep provider behavior visible
Pinned model IDs, first-party routes, disabled fallbacks, and disclosed task-specific response contracts.
Retry policy
One retry for 429, 5xx, or network failure—not wrong answers.
Operational metrics
Latency, failures, retries, and cost stay separate from accuracy.
Privacy
Published records omit keys and provider request identifiers.
Data and license
Download this edition
Model preflight ↓Run plan ↓Capture manifest ↓Results and individual scores ↓Raw retained attempts ↓Findings and failure inventory ↓
Repeato-created captures and aggregates are available under CC BY 4.0. Raw model output remains subject to provider terms. See the license notice.