DownloadDemo buchen

Repeato-Forschung · LLM-Vision-Benchmark

Kann man multimodalen LLMs UI-Tests anvertrauen?

Ein ausschließlich auf Screenshots basierender Benchmark für UI-Testautomatisierung: interaktive Elemente finden, das Fehlen eines Elements erkennen, sichtbare Layoutfehler identifizieren und die operativen Kosten einer Antwort erfassen.

2026-10-17

Sechs deterministische Screens. 24 Ziele zur Elementlokalisierung. 10 API-Konfigurationen. Drei Wiederholungen pro Fall. Ein gezielter Pilot, keine Rangliste.

Recherche von Stephan PetzlVeröffentlicht

Screens6
Grounding-Ziele24
Fehlerkategorien6
API-Konfigurationen10

Erkenntnisse aus dem Pilotprojekt

Was diese Ausgabe zeigt

Drei Wiederholungen pro Fall auf sechs synthetischen Screens zeigen Unterschiede, die weitere Untersuchungen verdienen. Sie begründen weder allgemeine Modellranglisten noch Zuverlässigkeit im Produktivbetrieb.

01

Bedienelemente präzise finden

GPT-6.1 Sol traf alle 72 angeforderten Ziele über drei Wiederholungen mit durchschnittlich 81,2 % Überlappung der Zielfläche. Vier Konfigurationen trafen jedes Ziel; die vorhergesagten Flächen unterschieden sich in ihrer Präzision.

02

Präsenzprüfungen erreichten die Obergrenze

Alle zehn Konfigurationen beantworteten sämtliche 36 Präsenzprüfungen korrekt. Dieser kleine Datensatz prüft die Zurückweisung nicht vorhandener Bedienelemente, unterscheidet die Modelle aber nicht nach Genauigkeit.

03

Einen Fehler zu bemerken ist erst der Anfang

Jev erkannte alle 36 intakten oder fehlerhaften Screens korrekt, benannte den genauen Fehler aber nur in 12 von 18 Versuchen mit fehlerhaften Screens. Dabei wurden sechs unterschiedliche Fehler jeweils dreimal geprüft.

Was das für die Testautomatisierung bedeutet

Verwende für jeden Testschritt die passende Art von Vision

Exakte Interaktion, kontinuierliche Beobachtung und semantische Interpretation stellen unterschiedliche Leistungsanforderungen. Es sollte nicht erwartet werden, dass eine einzige Computer-Vision-Technik alle drei Bereiche optimiert.

Lokale BeobachtungBis zu 20-mal pro SekundeAnfrage an ein entferntes ModellMedian 0,3–12 Sekunden in diesem PilotWie wir lokale Vision und LLMs kombinierenAnalyse einklappen

Repeatos lokale Bildverarbeitung kann den Screen mit 20 Hz prüfen, also alle 50 ms. Dieser Pilot maß je nach Aufgabe mediane Antwortzeiten entfernter Modelle von etwa 0,3 bis 12 Sekunden. Die gemeinsame Präsenzaufgabe erlaubt einen faireren Zeitvergleich; eine schnelle Klassifikation bedeutet keine ebenso schnelle Ziellokalisierung. Die lokalen und entfernten Werte stammen aus unterschiedlichen Messungen, nicht aus einem kontrollierten direkten Vergleich.

01

Schnelles Feedback bei wiederholten Interaktionen

Lokale visuelle Fingerabdrücke können kontinuierlich nach aufgezeichneten Zielen suchen und handeln, sobald der Bildschirm bereit ist. Remote-Modellanfragen in derselben Frequenz zu wiederholen, wäre unpraktisch.

02

Planbar, privat und kostengünstig

Lokales Matching erfordert weder einen Netzwerk-Roundtrip noch die Verfügbarkeit eines Anbieters und verursacht keine API-Kosten pro Anfrage. Bildschirme bleiben auf dem Arbeitsplatzrechner, und derselbe Algorithmus kann offline ausgeführt werden.

03

KI für Fragen, die Matching nicht beantworten kann

Die LLM-Bildverarbeitung spielt ihre Latenz aus, wenn ein Test eine semantische Beurteilung benötigt: Texte oder Zahlen interpretieren, dynamische Inhalte verstehen oder Konzepte wie einen Aufwärtstrend in einem Diagramm prüfen.

Praktische Aufteilung

Nutze lokale Computer Vision für die hochfrequente Steuerschleife – Ziele finden, warten, interagieren und visuell vergleichen. Wechsle gezielt zu KI-Vision, wenn die Prüfung tatsächlich Sprachverständnis oder semantisches Verständnis erfordert.

Experimente

Drei eigenständige UI-Vision-Aufgaben

Grounding, Elementvorhandensein und UI-Integrität beantworten unterschiedliche Fragen und verwenden separate Metriken. Der Pilot fasst sie nicht zu einer Gesamtbewertung zusammen.

Experiment 1 · UI-Grounding

Kann ein Modell das angeforderte UI-Element finden?

24 Ziele × drei Wiederholungen = 72 Versuche pro Konfiguration mit Koordinatenausgabe. Ungültige und fehlgeschlagene Antworten zählen als Fehlversuche.

Neun Konfigurationen können Zielkoordinaten liefern. GPT-6 Luna Decisions beantwortet Klassifikationsfragen und ist von den Ergebnissen zur Ziellokalisierung ausgenommen.

Box-IoUHöher ist besser · Skala von 0 bis 1
  1. GPT-6 Luna0.796
  2. GPT-6.1 Sol0.812
  3. Claude Sonnet 5.50.748
  4. Gemini 3.8 Flash0.733
  5. Grok 4.70.477
  6. Qwen3.8 Max (0902)0.760
  7. Muse Spark 1.30.630
  8. GPT-5.6 Luna0.786
  9. Jev 1.140.581
GesamtkostenNiedriger ist besser · 72 Anfragen
  1. GPT-6 Luna$0.0096
  2. GPT-6.1 Sol$0.1447
  3. Claude Sonnet 5.5$0.2970
  4. Gemini 3.8 Flash$0.1007
  5. Grok 4.7$0.6647
  6. Qwen3.8 Max (0902)$0.3254
  7. Muse Spark 1.3$0.2220
  8. GPT-5.6 Luna$0.0258
  9. Jev 1.14$0.0079
Mittlere LatenzNiedriger ist besser · pro Anfrage
  1. GPT-6 Luna2.35 s
  2. GPT-6.1 Sol2.91 s
  3. Claude Sonnet 5.52.32 s
  4. Gemini 3.8 Flash2.75 s
  5. Grok 4.716.91 s
  6. Qwen3.8 Max (0902)8.89 s
  7. Muse Spark 1.34.80 s
  8. GPT-5.6 Luna2.93 s
  9. Jev 1.140.37 s
Genauigkeit und Kosten im VergleichWeiter oben und weiter links ist besser.
Better value0.000.250.500.751.00$0.00$0.14$0.28$0.42$0.56$0.70Total cost for 72 requests (USD) →Box IoU →GPT-6 LunaGPT-6.1 SolClaude Sonnet 5.5Gemini 3.8 FlashGrok 4.7Qwen3.8 Max (0902)Muse Spark 1.3GPT-5.6 LunaJev 1.14
GPT-6 Lunaopenai/gpt-6-luna →
Bestanden99% Treffer0.7962.35 s$0.0096
GPT-6.1 Solopenai/gpt-6.1-sol →
Bestanden100 % Treffer0.8122.91 s$0.1447
Claude Sonnet 5.5anthropic/claude-sonnet-5.5 →
Ungültige Antwort97% Treffer0.7482.32 s$0.2970
Gemini 3.8 Flashgoogle/gemini-3.8-flash →
Bestanden100 % Treffer0.7332.75 s$0.1007
Grok 4.7x-ai/grok-4.7 →
Ungültige Antwort72% Treffer0.47716.91 s$0.6647
Qwen3.8 Max (0902)qwen/qwen3.8-max-0902 →
Bestanden100 % Treffer0.7608.89 s$0.3254
Muse Spark 1.3meta/muse-spark-1.3 →
Bestanden94% Treffer0.6304.80 s$0.2220
GPT-5.6 Lunaopenai/gpt-5.6-luna →
Bestanden99% Treffer0.7862.93 s$0.0258
Jev 1.14typesafe/jev-1.14 →
Bestanden100 % Treffer0.5810.37 s$0.0079

Methodik

So wird UI-Grounding bewertet

Vier Ziele pro Screenshot aus gängigen interaktiven UI-Elementtypen.

Punktgenauigkeit

Der vorhergesagte Klickpunkt liegt innerhalb der Zielbox.

Boxgenauigkeit

Mittlere IoU und Anteil der Vorhersagen mit IoU ≥ 0,5.

Gültigkeit

Anteil schemagültiger Antworten; drei Wiederholungen pro Fall zeigen die Konsistenz innerhalb eines Falls.

Experiment 2 · Elementvorhandensein

Kann ein Modell erkennen, wenn ein Element nicht vorhanden ist?

Sechs vorhandene und sechs fehlende Ziele, jeweils dreimal geprüft. Gemessen wird die sichtbare Präsenz, nicht das Erreichen eines Navigationsziels.

GPT-6 Lunaopenai/gpt-6-luna →
100 % gültig36/36 Fälle · 0 fehlgeschlagene Versuche100%100%100%0%1.50 s$0.0752pro 1.000 beibehaltene Fälle
GPT-6.1 Solopenai/gpt-6.1-sol →
100 % gültig36/36 Fälle · 0 fehlgeschlagene Versuche100%100%100%0%2.24 s$1.4487pro 1.000 beibehaltene Fälle
Claude Sonnet 5.5anthropic/claude-sonnet-5.5 →
100 % gültig36/36 Fälle · 0 fehlgeschlagene Versuche100%100%100%0%2.02 s$2.9955pro 1.000 beibehaltene Fälle
Gemini 3.8 Flashgoogle/gemini-3.8-flash →
100 % gültig36/36 Fälle · 0 fehlgeschlagene Versuche100%100%100%0%2.44 s$1.2008pro 1.000 beibehaltene Fälle
Grok 4.7x-ai/grok-4.7 →
100 % gültig36/36 Fälle · 0 fehlgeschlagene Versuche100%100%100%0%2.66 s$2.8797pro 1.000 beibehaltene Fälle
Qwen3.8 Max (0902)qwen/qwen3.8-max-0902 →
100 % gültig36/36 Fälle · 0 fehlgeschlagene Versuche100%100%100%0%4.81 s$2.6708pro 1.000 beibehaltene Fälle
Muse Spark 1.3meta/muse-spark-1.3 →
100 % gültig36/36 Fälle · 0 fehlgeschlagene Versuche100%100%100%0%2.47 s$2.0156pro 1.000 beibehaltene Fälle
GPT-5.6 Lunaopenai/gpt-5.6-luna →
100 % gültig36/36 Fälle · 0 fehlgeschlagene Versuche100%100%100%0%1.59 s$0.1533pro 1.000 beibehaltene Fälle
Jev 1.14typesafe/jev-1.14 →
100 % gültig36/36 Fälle · 0 fehlgeschlagene Versuche100%100%100%0%0.38 s$0.1069pro 1.000 beibehaltene Fälle
GPT-6 Luna Decisionsopenai/gpt-6-luna-decisions →
100 % gültig36/36 Fälle · 0 fehlgeschlagene Versuche100%100%100%0%0.39 s$0.1062pro 1.000 beibehaltene Fälle

Drei Wiederholungen pro Fall auf sechs synthetischen Screens. Die Kosten enthalten gemeldete Cache-Rabatte. Die Ergebnisse belegen weder Zuverlässigkeit im Produktivbetrieb noch allgemeine Modellranglisten.

Methodik

So wird das Elementvorhandensein bewertet

Pro fehlerfreiem Bildschirm eine bekanntermaßen vorhandene und eine plausible, aber nicht vorhandene Abfrage.

Erkennung

Die ausgewogene Genauigkeit gewichtet vorhandene und nicht vorhandene Fälle gleich.

Klassenverhalten

Recall, Spezifität und Falsch-Positiv-Rate zeigen die Fehler­richtung.

Nur boolesch

found=true oder found=false; kein Lokalisierungswert.

Experiment 3 · UI-Integrität

Kann ein Modell zwischen intakten und fehlerhaften Layouts unterscheiden?

Sechs fehlerfreie und sechs gezielt fehlerhafte Screens × drei Wiederholungen. Einen fehlerhaften Screen zu erkennen, die genaue Fehlerart zu benennen und den betroffenen Bereich zu finden, sind getrennte Ergebnisse.

Fehlererkennungsgenauigkeit und Kosten im VergleichWeiter oben und weiter links ist besser.
Better value0.000.250.500.751.00$0.00$2.00$4.00$6.00$8.00$10.00Cost per 1,000 retained defect cases (USD) →Balanced accuracy →GPT-6 LunaGPT-6.1 SolClaude Sonnet 5.5Gemini 3.8 FlashGrok 4.7Qwen3.8 Max (0902)Muse Spark 1.3GPT-5.6 LunaJev 1.14GPT-6 Luna Decisions
Fehlererkennungsgenauigkeit und Antwortzeit im VergleichWeiter oben und weiter links ist besser. Die Latenz ist die mittlere Antwortzeit pro gespeichertem Fehlerprüffall. Die reine Klassifikationsvariante Decisions liefert keine Bereiche.
Better value0.000.250.500.751.000.0 s3.6 s7.2 s10.8 s14.4 s18.0 sMean latency (seconds) →Balanced accuracy →GPT-6 LunaGPT-6.1 SolClaude Sonnet 5.5Gemini 3.8 FlashGrok 4.7Qwen3.8 Max (0902)Muse Spark 1.3GPT-5.6 LunaJev 1.14GPT-6 Luna Decisions
GPT-6 Lunaopenai/gpt-6-luna →
100 % gültig36/36 Fälle · 0 fehlgeschlagene Versuche92%83%100%56%3.04 s$0.1751pro 1.000 beibehaltene Fälle
GPT-6.1 Solopenai/gpt-6.1-sol →
100 % gültig36/36 Fälle · 0 fehlgeschlagene Versuche92%83%100%33%3.64 s$2.3176pro 1.000 beibehaltene Fälle
Claude Sonnet 5.5anthropic/claude-sonnet-5.5 →
83 % gültig36/36 Fälle · 0 fehlgeschlagene Versuche83%67%100%50%2.87 s$4.8735pro 1.000 beibehaltene Fälle
Gemini 3.8 Flashgoogle/gemini-3.8-flash →
97% gültig36/36 Fälle · 0 fehlgeschlagene Versuche78%100%56%72%7.06 s$3.3765pro 1.000 beibehaltene Fälle
Grok 4.7x-ai/grok-4.7 →
97% gültig36/36 Fälle · 0 fehlgeschlagene Versuche94%94%94%72%17.48 s$9.6872pro 1.000 beibehaltene Fälle
Qwen3.8 Max (0902)qwen/qwen3.8-max-0902 →
97% gültig36/36 Fälle · 0 fehlgeschlagene Versuche94%94%94%39%11.78 s$5.3957pro 1.000 beibehaltene Fälle
Muse Spark 1.3meta/muse-spark-1.3 →
100 % gültig36/36 Fälle · 0 fehlgeschlagene Versuche89%78%100%56%6.34 s$3.8791pro 1.000 beibehaltene Fälle
GPT-5.6 Lunaopenai/gpt-5.6-luna →
100 % gültig36/36 Fälle · 0 fehlgeschlagene Versuche92%83%100%67%2.88 s$0.3518pro 1.000 beibehaltene Fälle
Jev 1.14typesafe/jev-1.14 →
100 % gültig36/36 Fälle · 0 fehlgeschlagene Versuche100%100%100%67%0.37 s$0.1166pro 1.000 beibehaltene Fälle
GPT-6 Luna Decisionsopenai/gpt-6-luna-decisions →
100 % gültig36/36 Fälle · 0 fehlgeschlagene Versuche83%67%100%50%0.42 s$0.1291pro 1.000 beibehaltene Fälle

Drei Wiederholungen pro Fall auf sechs synthetischen Screens. Die Kosten enthalten gemeldete Cache-Rabatte. Die Ergebnisse belegen weder Zuverlässigkeit im Produktivbetrieb noch allgemeine Modellranglisten.

Die bestehende Fehlerbewertung setzt die Bereichsüberlappung bei falscher Kategorie auf null. Luna Decisions wird ausschließlich für Klassifikation bewertet und muss keinen Bereich liefern.

Methodik

So wird die UI-Integrität bewertet

Gepaarte fehlerfreie und fehlerhafte Szenen decken sechs objektive Arten von Layout-Fehlern ab.

Erkennung

Ausgewogene Genauigkeit, Recall, Spezifität und Falsch-Positiv-Rate.

Klassifizierung

Genauigkeit der Fehlerkategorie nach erfolgreicher Erkennung.

Lokalisierung

IoU der vorhergesagten Box im Vergleich zu browserbasierten Fehlerregionen.

Unveränderter Datensatz

Drei Desktop- und drei mobile Bildschirme

Jede Testvorrichtung verfügt über eine fehlerfreie Kontrolle und eine absichtlich fehlerhafte Variante. Die Aufnahmen nutzen feste Viewports, lokale Assets, stabile Zielgeometrie und exakte Grenzen für Pixeldifferenzen.

Commerce-Katalog ohne Fehler
Commerce-Katalog mit Container-Clipping
Commerce-KatalogDesktop-Web · Container-Clipping
Analytics-Dashboard ohne Fehler
Analytics-Dashboard mit Textüberlauf
Analytics-DashboardDesktop-Web · Textüberlauf
IoT-Kontrollzentrum ohne Fehler
IoT-Kontrollzentrum mit Elementüberlappung
IoT-SteuerzentraleDesktop-Web · Elementüberlappung
Mobile-Banking ohne Fehler
Mobile-Banking mit Viewport-Beschnitt
Mobile-BankingiOS-ähnlich · Viewport-Beschnitt
Bordkarte ohne Fehler
Bordkarte mit verdeckter Aktion
BordkarteiOS-ähnlich · Verdeckte Aktion
Smart Home ohne Fehler
Smart Home mit fehlerhaftem responsivem Reflow
Smart HomeAndroid-ähnlich · Fehlerhafter responsiver Reflow

Modelldossiers

Alle erfassten Fälle prüfen

Jede Wiederholung bleibt sichtbar, einschließlich falscher, ungültiger und fehlgeschlagener Versuche. Klappe ein Modell auf, um seine Fälle zu prüfen. Der Rohdaten-Download enthält die Vorhersagen und nativen Entscheidungen.

openaiGPT-6 LunaVollständige Benchmark-Ergebnisse →openaiGPT-6.1 SolVollständige Benchmark-Ergebnisse →anthropicClaude Sonnet 5.5Vollständige Benchmark-Ergebnisse →googleGemini 3.8 FlashVollständige Benchmark-Ergebnisse →x-aiGrok 4.7Vollständige Benchmark-Ergebnisse →qwenQwen3.8 Max (0902)Vollständige Benchmark-Ergebnisse →metaMuse Spark 1.3Vollständige Benchmark-Ergebnisse →openaiGPT-5.6 LunaVollständige Benchmark-Ergebnisse →typesafeJev 1.14Vollständige Benchmark-Ergebnisse →openaiGPT-6 Luna DecisionsVollständige Benchmark-Ergebnisse →
GPT-6 Luna · 144 Fallausführungen

Zurückgegebene Modellversionen: openai/gpt-6-luna.

Ziele mit demselben Treffer-/Fehlerergebnis in allen drei Wiederholungen: 23 / 24. Übereinstimmung kann auch bedeuten, dass ein Ziel jedes Mal verfehlt wird.

FallWiederholungErgebnisAusgangZeit
commerce-catalog-desktop/clean/grounding/search1erfolgreichZiel getroffen3.33 s
commerce-catalog-desktop/clean/grounding/add-product1erfolgreichZiel getroffen1.92 s
commerce-catalog-desktop/clean/grounding/catalog-query1erfolgreichZiel getroffen2.65 s
commerce-catalog-desktop/clean/grounding/stock-filter1erfolgreichZiel getroffen3.77 s
commerce-catalog-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.33 s
commerce-catalog-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 191, "y": 145, "width": 69, "height": 46 }, "description": "The page heading is clipped on the left, showing only “ducts” instead of the full title." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 191, "y": 145, "width": 69, "height": 46 }, "description": "The page heading is clipped on the left, showing only “ducts” instead of the full title." }3.98 s
commerce-catalog-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.55 s
commerce-catalog-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.28 s
analytics-dashboard-desktop/clean/grounding/audience-tab1erfolgreichZiel getroffen2.66 s
analytics-dashboard-desktop/clean/grounding/revenue-card1erfolgreichZiel getroffen2.42 s
analytics-dashboard-desktop/clean/grounding/search-field1erfolgreichZiel getroffen2.01 s
analytics-dashboard-desktop/clean/grounding/compare-toggle1erfolgreichZiel getroffen2.24 s
analytics-dashboard-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.99 s
analytics-dashboard-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 248, "width": 326, "height": 31 }, "description": "The revenue card heading is clipped at the card’s right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 248, "width": 326, "height": 31 }, "description": "The revenue card heading is clipped at the card’s right edge." }2.59 s
analytics-dashboard-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.57 s
analytics-dashboard-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.57 s
iot-control-desktop/clean/grounding/devices-nav1erfolgreichZiel getroffen2.21 s
iot-control-desktop/clean/grounding/restart-device1erfolgreichZiel getroffen1.98 s
iot-control-desktop/clean/grounding/auto-mode1erfolgreichZiel getroffen2.62 s
iot-control-desktop/clean/grounding/pump-card1erfolgreichZiel getroffen2.80 s
iot-control-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.04 s
iot-control-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 222, "height": 64 }, "description": "The maintenance notification overlaps the device table rows." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 222, "height": 64 }, "description": "The maintenance notification overlaps the device table rows." }3.62 s
iot-control-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.49 s
iot-control-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.57 s
banking-ios/clean/grounding/profile1erfolgreichZiel getroffen2.46 s
banking-ios/clean/grounding/transaction-search1erfolgreichZiel getroffen2.81 s
banking-ios/clean/grounding/hide-balance1erfolgreichZiel getroffen2.59 s
banking-ios/clean/grounding/transfer1erfolgreichZiel getroffen1.87 s
banking-ios/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.16 s
banking-ios/defect/layout-defect1erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }6.07 s
banking-ios/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.72 s
banking-ios/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.24 s
flight-pass-ios/clean/grounding/boarding-pass1erfolgreichZiel getroffen2.49 s
flight-pass-ios/clean/grounding/check-in1erfolgreichZiel getroffen2.78 s
flight-pass-ios/clean/grounding/trips-tab1erfolgreichZiel getroffen1.63 s
flight-pass-ios/clean/grounding/profile-tab1erfolgreichZiel getroffen2.53 s
flight-pass-ios/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.76 s
flight-pass-ios/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "A red flight-update banner covers the lower portion of the blue action button." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "A red flight-update banner covers the lower portion of the blue action button." }2.79 s
flight-pass-ios/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.24 s
flight-pass-ios/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.12 s
smart-home-android/clean/grounding/more1erfolgreichZiel getroffen1.47 s
smart-home-android/clean/grounding/room-search1erfolgreichZiel getroffen1.88 s
smart-home-android/clean/grounding/thermostat-card1erfolgreichZiel getroffen1.46 s
smart-home-android/clean/grounding/add-device1erfolgreichZiel getroffen1.38 s
smart-home-android/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.93 s
smart-home-android/defect/layout-defect1erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }4.04 s
smart-home-android/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.19 s
smart-home-android/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.21 s
commerce-catalog-desktop/clean/grounding/search2erfolgreichZiel getroffen2.62 s
commerce-catalog-desktop/clean/grounding/add-product2erfolgreichZiel getroffen2.12 s
commerce-catalog-desktop/clean/grounding/catalog-query2erfolgreichZiel getroffen1.54 s
commerce-catalog-desktop/clean/grounding/stock-filter2erfolgreichZiel getroffen2.71 s
commerce-catalog-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.88 s
commerce-catalog-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 151, "width": 68, "height": 44 }, "description": "The page heading is clipped on the left; only “ducts” is visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 151, "width": 68, "height": 44 }, "description": "The page heading is clipped on the left; only “ducts” is visible." }3.44 s
commerce-catalog-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.75 s
commerce-catalog-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.65 s
analytics-dashboard-desktop/clean/grounding/audience-tab2erfolgreichZiel getroffen3.18 s
analytics-dashboard-desktop/clean/grounding/revenue-card2erfolgreichZiel getroffen2.89 s
analytics-dashboard-desktop/clean/grounding/search-field2erfolgreichZiel getroffen2.21 s
analytics-dashboard-desktop/clean/grounding/compare-toggle2erfolgreichZiel getroffen2.67 s
analytics-dashboard-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.91 s
analytics-dashboard-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 253, "width": 326, "height": 37 }, "description": "The revenue card heading is cut off at the right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 253, "width": 326, "height": 37 }, "description": "The revenue card heading is cut off at the right edge." }2.65 s
analytics-dashboard-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.97 s
analytics-dashboard-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.67 s
iot-control-desktop/clean/grounding/devices-nav2erfolgreichZiel getroffen2.20 s
iot-control-desktop/clean/grounding/restart-device2erfolgreichZiel getroffen2.25 s
iot-control-desktop/clean/grounding/auto-mode2erfolgreichZiel getroffen2.86 s
iot-control-desktop/clean/grounding/pump-card2erfolgreichZiel getroffen3.25 s
iot-control-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.92 s
iot-control-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 222, "height": 64 }, "description": "The maintenance notification overlaps the device table rows." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 222, "height": 64 }, "description": "The maintenance notification overlaps the device table rows." }3.14 s
iot-control-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.77 s
iot-control-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.64 s
banking-ios/clean/grounding/profile2erfolgreichZiel getroffen3.40 s
banking-ios/clean/grounding/transaction-search2erfolgreichZiel getroffen2.17 s
banking-ios/clean/grounding/hide-balance2erfolgreichZiel getroffen2.38 s
banking-ios/clean/grounding/transfer2erfolgreichZiel getroffen1.75 s
banking-ios/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.87 s
banking-ios/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 72, "width": 240, "height": 32 }, "description": "The first letter of the Banking heading is clipped at the left edge of its container." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 72, "width": 240, "height": 32 }, "description": "The first letter of the Banking heading is clipped at the left edge of its container." }5.97 s
banking-ios/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.17 s
banking-ios/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.45 s
flight-pass-ios/clean/grounding/boarding-pass2erfolgreichZiel getroffen2.26 s
flight-pass-ios/clean/grounding/check-in2erfolgreichZiel getroffen1.47 s
flight-pass-ios/clean/grounding/trips-tab2erfolgreichZiel getroffen1.62 s
flight-pass-ios/clean/grounding/profile-tab2erfolgreichZiel verfehlt1.55 s
flight-pass-ios/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.20 s
flight-pass-ios/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner obscures part of the blue actionable button beneath it." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner obscures part of the blue actionable button beneath it." }3.68 s
flight-pass-ios/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.22 s
flight-pass-ios/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.09 s
smart-home-android/clean/grounding/more2erfolgreichZiel getroffen1.72 s
smart-home-android/clean/grounding/room-search2erfolgreichZiel getroffen2.22 s
smart-home-android/clean/grounding/thermostat-card2erfolgreichZiel getroffen1.53 s
smart-home-android/clean/grounding/add-device2erfolgreichZiel getroffen1.48 s
smart-home-android/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.82 s
smart-home-android/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 354, "y": 510, "width": 646, "height": 69 }, "description": "The second Quick view card extends beyond the right edge of the viewport and is cropped." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 354, "y": 510, "width": 646, "height": 69 }, "description": "The second Quick view card extends beyond the right edge of the viewport and is cropped." }4.63 s
smart-home-android/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.18 s
smart-home-android/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.45 s
commerce-catalog-desktop/clean/grounding/search3erfolgreichZiel getroffen2.81 s
commerce-catalog-desktop/clean/grounding/add-product3erfolgreichZiel getroffen1.70 s
commerce-catalog-desktop/clean/grounding/catalog-query3erfolgreichZiel getroffen2.45 s
commerce-catalog-desktop/clean/grounding/stock-filter3erfolgreichZiel getroffen3.83 s
commerce-catalog-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }3.66 s
commerce-catalog-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 191, "y": 150, "width": 70, "height": 40 }, "description": "The page title is visibly clipped at its beginning, showing only “ducts”." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 191, "y": 150, "width": 70, "height": 40 }, "description": "The page title is visibly clipped at its beginning, showing only “ducts”." }3.67 s
commerce-catalog-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.85 s
commerce-catalog-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.33 s
analytics-dashboard-desktop/clean/grounding/audience-tab3erfolgreichZiel getroffen3.45 s
analytics-dashboard-desktop/clean/grounding/revenue-card3erfolgreichZiel getroffen2.06 s
analytics-dashboard-desktop/clean/grounding/search-field3erfolgreichZiel getroffen2.00 s
analytics-dashboard-desktop/clean/grounding/compare-toggle3erfolgreichZiel getroffen2.30 s
analytics-dashboard-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.75 s
analytics-dashboard-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 253, "width": 326, "height": 32 }, "description": "The monthly revenue card heading is clipped at the card’s right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 253, "width": 326, "height": 32 }, "description": "The monthly revenue card heading is clipped at the card’s right edge." }3.12 s
analytics-dashboard-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.27 s
analytics-dashboard-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.32 s
iot-control-desktop/clean/grounding/devices-nav3erfolgreichZiel getroffen2.50 s
iot-control-desktop/clean/grounding/restart-device3erfolgreichZiel getroffen2.75 s
iot-control-desktop/clean/grounding/auto-mode3erfolgreichZiel getroffen2.79 s
iot-control-desktop/clean/grounding/pump-card3erfolgreichZiel getroffen2.32 s
iot-control-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }3.03 s
iot-control-desktop/defect/layout-defect3erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }3.08 s
iot-control-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.44 s
iot-control-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.71 s
banking-ios/clean/grounding/profile3erfolgreichZiel getroffen2.41 s
banking-ios/clean/grounding/transaction-search3erfolgreichZiel getroffen2.90 s
banking-ios/clean/grounding/hide-balance3erfolgreichZiel getroffen2.29 s
banking-ios/clean/grounding/transfer3erfolgreichZiel getroffen2.51 s
banking-ios/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.21 s
banking-ios/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 0, "y": 69, "width": 285, "height": 35 }, "description": "The heading at the upper left is clipped on its left side." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 0, "y": 69, "width": 285, "height": 35 }, "description": "The heading at the upper left is clipped on its left side." }5.43 s
banking-ios/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.45 s
banking-ios/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.26 s
flight-pass-ios/clean/grounding/boarding-pass3erfolgreichZiel getroffen2.16 s
flight-pass-ios/clean/grounding/check-in3erfolgreichZiel getroffen2.23 s
flight-pass-ios/clean/grounding/trips-tab3erfolgreichZiel getroffen2.66 s
flight-pass-ios/clean/grounding/profile-tab3erfolgreichZiel getroffen3.25 s
flight-pass-ios/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.77 s
flight-pass-ios/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner obscures part of the blue actionable button beneath it." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner obscures part of the blue actionable button beneath it." }2.89 s
flight-pass-ios/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.24 s
flight-pass-ios/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.06 s
smart-home-android/clean/grounding/more3erfolgreichZiel getroffen2.34 s
smart-home-android/clean/grounding/room-search3erfolgreichZiel getroffen2.18 s
smart-home-android/clean/grounding/thermostat-card3erfolgreichZiel getroffen1.72 s
smart-home-android/clean/grounding/add-device3erfolgreichZiel getroffen1.51 s
smart-home-android/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.43 s
smart-home-android/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 354, "y": 510, "width": 646, "height": 69 }, "description": "The Kitchen lights card is cut off at the right edge of the viewport." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 354, "y": 510, "width": 646, "height": 69 }, "description": "The Kitchen lights card is cut off at the right edge of the viewport." }5.87 s
smart-home-android/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.47 s
smart-home-android/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.66 s
GPT-6.1 Sol · 144 Fallausführungen

Zurückgegebene Modellversionen: openai/gpt-6.1-sol.

Ziele mit demselben Treffer-/Fehlerergebnis in allen drei Wiederholungen: 24 / 24. Übereinstimmung kann auch bedeuten, dass ein Ziel jedes Mal verfehlt wird.

FallWiederholungErgebnisAusgangZeit
commerce-catalog-desktop/clean/grounding/search1erfolgreichZiel getroffen3.73 s
commerce-catalog-desktop/clean/grounding/add-product1erfolgreichZiel getroffen3.41 s
commerce-catalog-desktop/clean/grounding/catalog-query1erfolgreichZiel getroffen2.75 s
commerce-catalog-desktop/clean/grounding/stock-filter1erfolgreichZiel getroffen3.00 s
commerce-catalog-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }3.79 s
commerce-catalog-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 68, "height": 68 }, "description": "The page heading and its small label are clipped on the left, leaving only trailing portions of the text visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 68, "height": 68 }, "description": "The page heading and its small label are clipped on the left, leaving only trailing portions of the text visible." }5.65 s
commerce-catalog-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.04 s
commerce-catalog-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.52 s
analytics-dashboard-desktop/clean/grounding/audience-tab1erfolgreichZiel getroffen4.02 s
analytics-dashboard-desktop/clean/grounding/revenue-card1erfolgreichZiel getroffen3.04 s
analytics-dashboard-desktop/clean/grounding/search-field1erfolgreichZiel getroffen3.42 s
analytics-dashboard-desktop/clean/grounding/compare-toggle1erfolgreichZiel getroffen3.78 s
analytics-dashboard-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.68 s
analytics-dashboard-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 250, "width": 326, "height": 31 }, "description": "The monthly revenue card heading is clipped at the card's right edge." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 250, "width": 326, "height": 31 }, "description": "The monthly revenue card heading is clipped at the card's right edge." }3.75 s
analytics-dashboard-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.18 s
analytics-dashboard-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.13 s
iot-control-desktop/clean/grounding/devices-nav1erfolgreichZiel getroffen3.30 s
iot-control-desktop/clean/grounding/restart-device1erfolgreichZiel getroffen2.72 s
iot-control-desktop/clean/grounding/auto-mode1erfolgreichZiel getroffen2.67 s
iot-control-desktop/clean/grounding/pump-card1erfolgreichZiel getroffen2.61 s
iot-control-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.52 s
iot-control-desktop/defect/layout-defect1erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.67 s
iot-control-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.46 s
iot-control-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.27 s
banking-ios/clean/grounding/profile1erfolgreichZiel getroffen2.88 s
banking-ios/clean/grounding/transaction-search1erfolgreichZiel getroffen2.62 s
banking-ios/clean/grounding/hide-balance1erfolgreichZiel getroffen2.78 s
banking-ios/clean/grounding/transfer1erfolgreichZiel getroffen3.09 s
banking-ios/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.96 s
banking-ios/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 47, "width": 238, "height": 59 }, "description": "The brand label and Banking heading are clipped along their left edge, cutting off their initial letters." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 47, "width": 238, "height": 59 }, "description": "The brand label and Banking heading are clipped along their left edge, cutting off their initial letters." }8.64 s
banking-ios/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.87 s
banking-ios/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.35 s
flight-pass-ios/clean/grounding/boarding-pass1erfolgreichZiel getroffen2.48 s
flight-pass-ios/clean/grounding/check-in1erfolgreichZiel getroffen4.77 s
flight-pass-ios/clean/grounding/trips-tab1erfolgreichZiel getroffen2.43 s
flight-pass-ios/clean/grounding/profile-tab1erfolgreichZiel getroffen2.85 s
flight-pass-ios/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.39 s
flight-pass-ios/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 64 }, "description": "The flight update banner overlaps the bottom blue action button, obscuring its label and lower portion." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 64 }, "description": "The flight update banner overlaps the bottom blue action button, obscuring its label and lower portion." }3.86 s
flight-pass-ios/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.98 s
flight-pass-ios/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.53 s
smart-home-android/clean/grounding/more1erfolgreichZiel getroffen2.27 s
smart-home-android/clean/grounding/room-search1erfolgreichZiel getroffen2.44 s
smart-home-android/clean/grounding/thermostat-card1erfolgreichZiel getroffen2.78 s
smart-home-android/clean/grounding/add-device1erfolgreichZiel getroffen2.72 s
smart-home-android/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.44 s
smart-home-android/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 70 }, "description": "The Kitchen lights card is shifted right and clipped at the list's right edge, cutting off its right side and trailing control." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 70 }, "description": "The Kitchen lights card is shifted right and clipped at the list's right edge, cutting off its right side and trailing control." }5.47 s
smart-home-android/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.67 s
smart-home-android/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.36 s
commerce-catalog-desktop/clean/grounding/search2erfolgreichZiel getroffen2.80 s
commerce-catalog-desktop/clean/grounding/add-product2erfolgreichZiel getroffen2.61 s
commerce-catalog-desktop/clean/grounding/catalog-query2erfolgreichZiel getroffen2.82 s
commerce-catalog-desktop/clean/grounding/stock-filter2erfolgreichZiel getroffen3.15 s
commerce-catalog-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }3.00 s
commerce-catalog-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 73, "height": 68 }, "description": "The page heading and its section label are clipped on the left, hiding their initial characters." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 73, "height": 68 }, "description": "The page heading and its section label are clipped on the left, hiding their initial characters." }4.90 s
commerce-catalog-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.41 s
commerce-catalog-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.60 s
analytics-dashboard-desktop/clean/grounding/audience-tab2erfolgreichZiel getroffen2.80 s
analytics-dashboard-desktop/clean/grounding/revenue-card2erfolgreichZiel getroffen2.48 s
analytics-dashboard-desktop/clean/grounding/search-field2erfolgreichZiel getroffen2.85 s
analytics-dashboard-desktop/clean/grounding/compare-toggle2erfolgreichZiel getroffen3.92 s
analytics-dashboard-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.98 s
analytics-dashboard-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 249, "width": 326, "height": 32 }, "description": "The monthly revenue heading is cut off at the right edge of its card." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 249, "width": 326, "height": 32 }, "description": "The monthly revenue heading is cut off at the right edge of its card." }3.47 s
analytics-dashboard-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.60 s
analytics-dashboard-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.42 s
iot-control-desktop/clean/grounding/devices-nav2erfolgreichZiel getroffen3.34 s
iot-control-desktop/clean/grounding/restart-device2erfolgreichZiel getroffen2.94 s
iot-control-desktop/clean/grounding/auto-mode2erfolgreichZiel getroffen4.14 s
iot-control-desktop/clean/grounding/pump-card2erfolgreichZiel getroffen2.51 s
iot-control-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.47 s
iot-control-desktop/defect/layout-defect2erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.90 s
iot-control-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.94 s
iot-control-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.50 s
banking-ios/clean/grounding/profile2erfolgreichZiel getroffen2.89 s
banking-ios/clean/grounding/transaction-search2erfolgreichZiel getroffen2.58 s
banking-ios/clean/grounding/hide-balance2erfolgreichZiel getroffen2.40 s
banking-ios/clean/grounding/transfer2erfolgreichZiel getroffen2.81 s
banking-ios/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.89 s
banking-ios/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 69, "width": 241, "height": 38 }, "description": "The left edge of the Banking heading is clipped, cutting off part of its first letter." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 69, "width": 241, "height": 38 }, "description": "The left edge of the Banking heading is clipped, cutting off part of its first letter." }7.42 s
banking-ios/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.72 s
banking-ios/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.02 s
flight-pass-ios/clean/grounding/boarding-pass2erfolgreichZiel getroffen3.14 s
flight-pass-ios/clean/grounding/check-in2erfolgreichZiel getroffen2.86 s
flight-pass-ios/clean/grounding/trips-tab2erfolgreichZiel getroffen2.24 s
flight-pass-ios/clean/grounding/profile-tab2erfolgreichZiel getroffen2.47 s
flight-pass-ios/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.99 s
flight-pass-ios/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 64 }, "description": "The red flight-update banner overlaps the blue bottom action button, obscuring its label and much of its clickable area." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 64 }, "description": "The red flight-update banner overlaps the blue bottom action button, obscuring its label and much of its clickable area." }4.43 s
flight-pass-ios/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.94 s
flight-pass-ios/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.89 s
smart-home-android/clean/grounding/more2erfolgreichZiel getroffen2.35 s
smart-home-android/clean/grounding/room-search2erfolgreichZiel getroffen2.58 s
smart-home-android/clean/grounding/thermostat-card2erfolgreichZiel getroffen2.82 s
smart-home-android/clean/grounding/add-device2erfolgreichZiel getroffen2.71 s
smart-home-android/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.13 s
smart-home-android/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 71 }, "description": "The Kitchen lights row is shifted right and clipped at the content boundary, cutting off its right edge and trailing control." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 71 }, "description": "The Kitchen lights row is shifted right and clipped at the content boundary, cutting off its right edge and trailing control." }5.02 s
smart-home-android/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.05 s
smart-home-android/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.98 s
commerce-catalog-desktop/clean/grounding/search3erfolgreichZiel getroffen3.93 s
commerce-catalog-desktop/clean/grounding/add-product3erfolgreichZiel getroffen2.40 s
commerce-catalog-desktop/clean/grounding/catalog-query3erfolgreichZiel getroffen3.39 s
commerce-catalog-desktop/clean/grounding/stock-filter3erfolgreichZiel getroffen2.85 s
commerce-catalog-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the specified categories." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the specified categories." }3.40 s
commerce-catalog-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 70, "height": 68 }, "description": "The page heading and its overline are clipped on the left, leaving only trailing portions of the text visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 70, "height": 68 }, "description": "The page heading and its overline are clipped on the left, leaving only trailing portions of the text visible." }4.58 s
commerce-catalog-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.23 s
commerce-catalog-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }3.35 s
analytics-dashboard-desktop/clean/grounding/audience-tab3erfolgreichZiel getroffen2.94 s
analytics-dashboard-desktop/clean/grounding/revenue-card3erfolgreichZiel getroffen2.86 s
analytics-dashboard-desktop/clean/grounding/search-field3erfolgreichZiel getroffen2.81 s
analytics-dashboard-desktop/clean/grounding/compare-toggle3erfolgreichZiel getroffen3.48 s
analytics-dashboard-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.59 s
analytics-dashboard-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 249, "width": 327, "height": 32 }, "description": "The monthly revenue heading is cut off at the right edge of its card." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 249, "width": 327, "height": 32 }, "description": "The monthly revenue heading is cut off at the right edge of its card." }3.59 s
analytics-dashboard-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.55 s
analytics-dashboard-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.03 s
iot-control-desktop/clean/grounding/devices-nav3erfolgreichZiel getroffen3.04 s
iot-control-desktop/clean/grounding/restart-device3erfolgreichZiel getroffen3.26 s
iot-control-desktop/clean/grounding/auto-mode3erfolgreichZiel getroffen2.99 s
iot-control-desktop/clean/grounding/pump-card3erfolgreichZiel getroffen2.80 s
iot-control-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }3.12 s
iot-control-desktop/defect/layout-defect3erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.84 s
iot-control-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.39 s
iot-control-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.19 s
banking-ios/clean/grounding/profile3erfolgreichZiel getroffen3.35 s
banking-ios/clean/grounding/transaction-search3erfolgreichZiel getroffen2.37 s
banking-ios/clean/grounding/hide-balance3erfolgreichZiel getroffen3.37 s
banking-ios/clean/grounding/transfer3erfolgreichZiel getroffen2.54 s
banking-ios/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.05 s
banking-ios/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 44, "y": 68, "width": 245, "height": 39 }, "description": "The left edge of the Banking heading is clipped, cutting off part of its first letter." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 44, "y": 68, "width": 245, "height": 39 }, "description": "The left edge of the Banking heading is clipped, cutting off part of its first letter." }6.93 s
banking-ios/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.85 s
banking-ios/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.60 s
flight-pass-ios/clean/grounding/boarding-pass3erfolgreichZiel getroffen2.12 s
flight-pass-ios/clean/grounding/check-in3erfolgreichZiel getroffen2.21 s
flight-pass-ios/clean/grounding/trips-tab3erfolgreichZiel getroffen3.53 s
flight-pass-ios/clean/grounding/profile-tab3erfolgreichZiel getroffen2.35 s
flight-pass-ios/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.03 s
flight-pass-ios/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner covers the lower portion and label of the blue action button above the bottom navigation." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner covers the lower portion and label of the blue action button above the bottom navigation." }4.19 s
flight-pass-ios/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.91 s
flight-pass-ios/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.12 s
smart-home-android/clean/grounding/more3erfolgreichZiel getroffen2.30 s
smart-home-android/clean/grounding/room-search3erfolgreichZiel getroffen2.60 s
smart-home-android/clean/grounding/thermostat-card3erfolgreichZiel getroffen2.75 s
smart-home-android/clean/grounding/add-device3erfolgreichZiel getroffen2.49 s
smart-home-android/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.17 s
smart-home-android/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 70 }, "description": "The Kitchen lights card is shifted right and clipped at the list container’s right edge, cutting off its right side and trailing control." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 70 }, "description": "The Kitchen lights card is shifted right and clipped at the list container’s right edge, cutting off its right side and trailing control." }5.17 s
smart-home-android/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.06 s
smart-home-android/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.07 s
Claude Sonnet 5.5 · 144 Fallausführungen

Zurückgegebene Modellversionen: anthropic/claude-sonnet-5.5.

Ziele mit demselben Treffer-/Fehlerergebnis in allen drei Wiederholungen: 22 / 24. Übereinstimmung kann auch bedeuten, dass ein Ziel jedes Mal verfehlt wird.

FallWiederholungErgebnisAusgangZeit
commerce-catalog-desktop/clean/grounding/search1erfolgreichZiel getroffen3.96 s
commerce-catalog-desktop/clean/grounding/add-product1ungültige AntwortZiel verfehlt2.02 s
commerce-catalog-desktop/clean/grounding/catalog-query1erfolgreichZiel getroffen3.08 s
commerce-catalog-desktop/clean/grounding/stock-filter1erfolgreichZiel getroffen2.40 s
commerce-catalog-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact and aligned." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact and aligned." }3.75 s
commerce-catalog-desktop/defect/layout-defect1ungültige AntwortThe answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 191, "y": 123, "width": 200, "height": 75 }, "description": "The page heading and eyebrow text are cut off on the left edge. The eyebrow reads 'IANAGEMENT' instead of 'MANAGEMENT', and the title reads 'ducts' instead of 'Products', so the leading characters are clipped by the container." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 191, "y": 123, "width": 200, "height": 75 }, "description": "The page heading and eyebrow text are cut off on the left edge. The eyebrow reads 'IANAGEMENT' instead of 'MANAGEMENT', and the title reads 'ducts' instead of 'Products', so the leading characters are clipped by the container." }4.65 s
commerce-catalog-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.46 s
commerce-catalog-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.24 s
analytics-dashboard-desktop/clean/grounding/audience-tab1erfolgreichZiel getroffen2.47 s
analytics-dashboard-desktop/clean/grounding/revenue-card1erfolgreichZiel getroffen2.46 s
analytics-dashboard-desktop/clean/grounding/search-field1erfolgreichZiel getroffen2.94 s
analytics-dashboard-desktop/clean/grounding/compare-toggle1erfolgreichZiel getroffen2.31 s
analytics-dashboard-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible clipping, overflow, overlap, obscured controls, or reflow problems; all text and controls are fully visible and properly contained." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible clipping, overflow, overlap, obscured controls, or reflow problems; all text and controls are fully visible and properly contained." }4.48 s
analytics-dashboard-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 250, "width": 356, "height": 32 }, "description": "The red card title 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is clipped by the container." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 250, "width": 356, "height": 32 }, "description": "The red card title 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is clipped by the container." }2.55 s
analytics-dashboard-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.14 s
analytics-dashboard-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.14 s
iot-control-desktop/clean/grounding/devices-nav1erfolgreichZiel getroffen3.05 s
iot-control-desktop/clean/grounding/restart-device1erfolgreichZiel getroffen2.13 s
iot-control-desktop/clean/grounding/auto-mode1erfolgreichZiel getroffen2.39 s
iot-control-desktop/clean/grounding/pump-card1erfolgreichZiel getroffen2.31 s
iot-control-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact." }3.98 s
iot-control-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' banner floats over the table rows, covering part of the Cooling pump and Intake valve row areas and overlapping the row divider." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' banner floats over the table rows, covering part of the Cooling pump and Intake valve row areas and overlapping the row divider." }4.73 s
iot-control-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.99 s
iot-control-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.35 s
banking-ios/clean/grounding/profile1erfolgreichZiel getroffen1.84 s
banking-ios/clean/grounding/transaction-search1erfolgreichZiel getroffen3.14 s
banking-ios/clean/grounding/hide-balance1erfolgreichZiel getroffen1.92 s
banking-ios/clean/grounding/transfer1erfolgreichZiel getroffen1.82 s
banking-ios/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all text and controls are fully visible, contained, and not overlapping." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all text and controls are fully visible, contained, and not overlapping." }2.21 s
banking-ios/defect/layout-defect1ungültige AntwortThe answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 300, "height": 115 }, "description": "The header title is cut off at the left screen edge: 'banking' renders as 'oanking' and the 'MAYA' label above it as 'IAYA', so the leading characters are cropped." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 300, "height": 115 }, "description": "The header title is cut off at the left screen edge: 'banking' renders as 'oanking' and the 'MAYA' label above it as 'IAYA', so the leading characters are cropped." }3.40 s
banking-ios/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.51 s
banking-ios/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.83 s
flight-pass-ios/clean/grounding/boarding-pass1erfolgreichZiel getroffen2.46 s
flight-pass-ios/clean/grounding/check-in1erfolgreichZiel getroffen1.91 s
flight-pass-ios/clean/grounding/trips-tab1erfolgreichZiel getroffen1.89 s
flight-pass-ios/clean/grounding/profile-tab1erfolgreichZiel getroffen2.30 s
flight-pass-ios/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; content is fully visible, text fits its containers, and no elements overlap or obscure controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; content is fully visible, text fits its containers, and no elements overlap or obscure controls." }2.34 s
flight-pass-ios/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 80 }, "description": "A red 'Flight update' banner covers the blue primary button near the bottom, hiding most of its label so only the tops of the letters remain visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 80 }, "description": "A red 'Flight update' banner covers the blue primary button near the bottom, hiding most of its label so only the tops of the letters remain visible." }3.21 s
flight-pass-ios/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.77 s
flight-pass-ios/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }4.33 s
smart-home-android/clean/grounding/more1erfolgreichZiel getroffen2.23 s
smart-home-android/clean/grounding/room-search1erfolgreichZiel getroffen9.81 s
smart-home-android/clean/grounding/thermostat-card1erfolgreichZiel getroffen2.25 s
smart-home-android/clean/grounding/add-device1erfolgreichZiel getroffen1.83 s
smart-home-android/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; all elements are fully rendered and aligned." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; all elements are fully rendered and aligned." }4.38 s
smart-home-android/defect/layout-defect1ungültige AntwortThe answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 533, "width": 646, "height": 46 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its right side is cut off." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 533, "width": 646, "height": 46 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its right side is cut off." }5.42 s
smart-home-android/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.53 s
smart-home-android/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.99 s
commerce-catalog-desktop/clean/grounding/search2erfolgreichZiel getroffen2.13 s
commerce-catalog-desktop/clean/grounding/add-product2erfolgreichZiel getroffen1.89 s
commerce-catalog-desktop/clean/grounding/catalog-query2erfolgreichZiel getroffen2.72 s
commerce-catalog-desktop/clean/grounding/stock-filter2erfolgreichZiel getroffen2.05 s
commerce-catalog-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." }2.41 s
commerce-catalog-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 125, "width": 205, "height": 57 }, "description": "The page heading and eyebrow text are cut off at the left edge of their container, showing 'IANAGEMENT' and 'ducts' instead of the full words." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 125, "width": 205, "height": 57 }, "description": "The page heading and eyebrow text are cut off at the left edge of their container, showing 'IANAGEMENT' and 'ducts' instead of the full words." }3.99 s
commerce-catalog-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.64 s
commerce-catalog-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.47 s
analytics-dashboard-desktop/clean/grounding/audience-tab2erfolgreichZiel getroffen2.29 s
analytics-dashboard-desktop/clean/grounding/revenue-card2erfolgreichZiel getroffen2.15 s
analytics-dashboard-desktop/clean/grounding/search-field2erfolgreichZiel getroffen2.47 s
analytics-dashboard-desktop/clean/grounding/compare-toggle2erfolgreichZiel getroffen2.02 s
analytics-dashboard-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible. The layout appears intact." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible. The layout appears intact." }3.55 s
analytics-dashboard-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 249, "width": 355, "height": 40 }, "description": "The red heading 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is clipped by the container." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 249, "width": 355, "height": 40 }, "description": "The red heading 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is clipped by the container." }2.22 s
analytics-dashboard-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.77 s
analytics-dashboard-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.68 s
iot-control-desktop/clean/grounding/devices-nav2erfolgreichZiel getroffen2.64 s
iot-control-desktop/clean/grounding/restart-device2erfolgreichZiel getroffen2.20 s
iot-control-desktop/clean/grounding/auto-mode2erfolgreichZiel getroffen2.46 s
iot-control-desktop/clean/grounding/pump-card2erfolgreichZiel getroffen2.21 s
iot-control-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." }2.09 s
iot-control-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "A red 'Maintenance scheduled' banner floats over the device table, covering parts of the Cooling pump and Intake valve rows and overlapping the row divider." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "A red 'Maintenance scheduled' banner floats over the device table, covering parts of the Cooling pump and Intake valve rows and overlapping the row divider." }3.48 s
iot-control-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.35 s
iot-control-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.89 s
banking-ios/clean/grounding/profile2erfolgreichZiel getroffen2.23 s
banking-ios/clean/grounding/transaction-search2erfolgreichZiel getroffen1.61 s
banking-ios/clean/grounding/hide-balance2erfolgreichZiel getroffen1.87 s
banking-ios/clean/grounding/transfer2erfolgreichZiel getroffen1.83 s
banking-ios/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all text, cards, and controls render fully within their containers without overlap or cropping." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all text, cards, and controls render fully within their containers without overlap or cropping." }2.29 s
banking-ios/defect/layout-defect2ungültige AntwortThe answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 300, "height": 55 }, "description": "The header text at the top left is cut off by the left edge of the viewport: the brand label reads 'IAYA' and the title reads 'anking' instead of 'Banking', so the first letters are cropped." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 300, "height": 55 }, "description": "The header text at the top left is cut off by the left edge of the viewport: the brand label reads 'IAYA' and the title reads 'anking' instead of 'Banking', so the first letters are cropped." }2.89 s
banking-ios/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.61 s
banking-ios/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.63 s
flight-pass-ios/clean/grounding/boarding-pass2erfolgreichZiel getroffen2.21 s
flight-pass-ios/clean/grounding/check-in2erfolgreichZiel getroffen2.25 s
flight-pass-ios/clean/grounding/trips-tab2erfolgreichZiel getroffen1.87 s
flight-pass-ios/clean/grounding/profile-tab2erfolgreichZiel getroffen2.24 s
flight-pass-ios/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; content is fully within the viewport, text fits its containers, and no elements overlap or obscure controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; content is fully within the viewport, text fits its containers, and no elements overlap or obscure controls." }2.20 s
flight-pass-ios/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 82 }, "description": "A red 'Flight update' banner overlaps the blue check-in button, hiding its label so the button text is cut off and the control is partially obscured." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 82 }, "description": "A red 'Flight update' banner overlaps the blue check-in button, hiding its label so the button text is cut off and the control is partially obscured." }2.08 s
flight-pass-ios/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.80 s
flight-pass-ios/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.71 s
smart-home-android/clean/grounding/more2erfolgreichZiel getroffen1.91 s
smart-home-android/clean/grounding/room-search2erfolgreichZiel getroffen1.75 s
smart-home-android/clean/grounding/thermostat-card2erfolgreichZiel getroffen2.58 s
smart-home-android/clean/grounding/add-device2erfolgreichZiel getroffen1.76 s
smart-home-android/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, elements are not overlapping, and nothing is cropped at the viewport edges." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, elements are not overlapping, and nothing is cropped at the viewport edges." }1.77 s
smart-home-android/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 531, "width": 628, "height": 55 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its chevron is cut off." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 531, "width": 628, "height": 55 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its chevron is cut off." }2.13 s
smart-home-android/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.74 s
smart-home-android/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.58 s
commerce-catalog-desktop/clean/grounding/search3erfolgreichZiel getroffen2.58 s
commerce-catalog-desktop/clean/grounding/add-product3erfolgreichZiel getroffen1.92 s
commerce-catalog-desktop/clean/grounding/catalog-query3erfolgreichZiel getroffen3.00 s
commerce-catalog-desktop/clean/grounding/stock-filter3erfolgreichZiel getroffen2.02 s
commerce-catalog-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact and aligned." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact and aligned." }2.56 s
commerce-catalog-desktop/defect/layout-defect3ungültige AntwortThe answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 125, "width": 205, "height": 57 }, "description": "The page heading and eyebrow text are cut off at the left edge of their container. The eyebrow reads 'IANAGEMENT' instead of 'MANAGEMENT' and the title reads 'ducts' instead of 'Products'." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 125, "width": 205, "height": 57 }, "description": "The page heading and eyebrow text are cut off at the left edge of their container. The eyebrow reads 'IANAGEMENT' instead of 'MANAGEMENT' and the title reads 'ducts' instead of 'Products'." }3.32 s
commerce-catalog-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.68 s
commerce-catalog-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.86 s
analytics-dashboard-desktop/clean/grounding/audience-tab3erfolgreichZiel getroffen2.65 s
analytics-dashboard-desktop/clean/grounding/revenue-card3erfolgreichZiel getroffen1.85 s
analytics-dashboard-desktop/clean/grounding/search-field3erfolgreichZiel getroffen2.33 s
analytics-dashboard-desktop/clean/grounding/compare-toggle3erfolgreichZiel getroffen2.31 s
analytics-dashboard-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, nothing is cropped or overlapping, and controls are unobstructed." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, nothing is cropped or overlapping, and controls are unobstructed." }2.16 s
analytics-dashboard-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 199, "y": 243, "width": 347, "height": 50 }, "description": "The red heading 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is truncated by the container." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 199, "y": 243, "width": 347, "height": 50 }, "description": "The red heading 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is truncated by the container." }2.14 s
analytics-dashboard-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.05 s
analytics-dashboard-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.21 s
iot-control-desktop/clean/grounding/devices-nav3erfolgreichZiel getroffen2.43 s
iot-control-desktop/clean/grounding/restart-device3erfolgreichZiel getroffen2.00 s
iot-control-desktop/clean/grounding/auto-mode3erfolgreichZiel getroffen2.25 s
iot-control-desktop/clean/grounding/pump-card3erfolgreichZiel getroffen1.91 s
iot-control-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." }2.08 s
iot-control-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' banner floats over the table rows, covering part of the Cooling pump and Intake valve rows and breaking the table layout." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' banner floats over the table rows, covering part of the Cooling pump and Intake valve rows and breaking the table layout." }2.50 s
iot-control-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.88 s
iot-control-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.83 s
banking-ios/clean/grounding/profile3erfolgreichZiel getroffen1.75 s
banking-ios/clean/grounding/transaction-search3erfolgreichZiel getroffen2.71 s
banking-ios/clean/grounding/hide-balance3erfolgreichZiel getroffen1.76 s
banking-ios/clean/grounding/transfer3erfolgreichZiel getroffen1.72 s
banking-ios/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: text is fully contained, elements are aligned without overlap, and all controls are visible and unobstructed." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: text is fully contained, elements are aligned without overlap, and all controls are visible and unobstructed." }1.81 s
banking-ios/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 290, "height": 50 }, "description": "The header title 'Banking' and the brand label above it are cut off at the left edge of the screen, showing 'anking' and a partial 'MAYA' label." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 290, "height": 50 }, "description": "The header title 'Banking' and the brand label above it are cut off at the left edge of the screen, showing 'anking' and a partial 'MAYA' label." }1.91 s
banking-ios/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.70 s
banking-ios/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.92 s
flight-pass-ios/clean/grounding/boarding-pass3erfolgreichZiel getroffen2.36 s
flight-pass-ios/clean/grounding/check-in3erfolgreichZiel getroffen2.25 s
flight-pass-ios/clean/grounding/trips-tab3ungültige AntwortZiel verfehlt2.21 s
flight-pass-ios/clean/grounding/profile-tab3erfolgreichZiel getroffen1.73 s
flight-pass-ios/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: text, cards, button and tab bar are fully visible, aligned and not overlapping or clipped." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: text, cards, button and tab bar are fully visible, aligned and not overlapping or clipped." }2.12 s
flight-pass-ios/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 80 }, "description": "A red 'Flight update' banner overlaps the blue check-in button, covering most of its label so the button text is cut off and the control is partly hidden." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 80 }, "description": "A red 'Flight update' banner overlaps the blue check-in button, covering most of its label so the button text is cut off and the control is partly hidden." }2.08 s
flight-pass-ios/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.84 s
flight-pass-ios/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.54 s
smart-home-android/clean/grounding/more3erfolgreichZiel getroffen1.58 s
smart-home-android/clean/grounding/room-search3erfolgreichZiel getroffen1.86 s
smart-home-android/clean/grounding/thermostat-card3erfolgreichZiel getroffen2.24 s
smart-home-android/clean/grounding/add-device3erfolgreichZiel getroffen1.75 s
smart-home-android/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, elements do not overlap, and controls are fully visible and unobstructed." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, elements do not overlap, and controls are fully visible and unobstructed." }1.97 s
smart-home-android/defect/layout-defect3ungültige AntwortThe answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 525, "width": 636, "height": 96 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its right side is cut off." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 525, "width": 636, "height": 96 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its right side is cut off." }2.42 s
smart-home-android/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.67 s
smart-home-android/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.53 s
Gemini 3.8 Flash · 144 Fallausführungen

Zurückgegebene Modellversionen: google/gemini-3.8-flash.

Ziele mit demselben Treffer-/Fehlerergebnis in allen drei Wiederholungen: 24 / 24. Übereinstimmung kann auch bedeuten, dass ein Ziel jedes Mal verfehlt wird.

FallWiederholungErgebnisAusgangZeit
commerce-catalog-desktop/clean/grounding/search1erfolgreichZiel getroffen2.28 s
commerce-catalog-desktop/clean/grounding/add-product1erfolgreichZiel getroffen3.52 s
commerce-catalog-desktop/clean/grounding/catalog-query1erfolgreichZiel getroffen2.57 s
commerce-catalog-desktop/clean/grounding/stock-filter1erfolgreichZiel getroffen2.31 s
commerce-catalog-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }3.65 s
commerce-catalog-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 67, "height": 68 }, "description": "The page heading text is clipped on the left edge, cutting off 'Products' into 'ducts' and 'MANAGEMENT' into 'IANAGEMENT'." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 67, "height": 68 }, "description": "The page heading text is clipped on the left edge, cutting off 'Products' into 'ducts' and 'MANAGEMENT' into 'IANAGEMENT'." }4.67 s
commerce-catalog-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.46 s
commerce-catalog-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }3.09 s
analytics-dashboard-desktop/clean/grounding/audience-tab1erfolgreichZiel getroffen2.08 s
analytics-dashboard-desktop/clean/grounding/revenue-card1erfolgreichZiel getroffen2.17 s
analytics-dashboard-desktop/clean/grounding/search-field1erfolgreichZiel getroffen2.41 s
analytics-dashboard-desktop/clean/grounding/compare-toggle1erfolgreichZiel getroffen2.25 s
analytics-dashboard-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering issues detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering issues detected." }6.43 s
analytics-dashboard-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 202, "y": 251, "width": 326, "height": 28 }, "description": "The card heading text is clipped at the right boundary of its parent container." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 202, "y": 251, "width": 326, "height": 28 }, "description": "The card heading text is clipped at the right boundary of its parent container." }4.58 s
analytics-dashboard-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.44 s
analytics-dashboard-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.84 s
iot-control-desktop/clean/grounding/devices-nav1erfolgreichZiel getroffen2.46 s
iot-control-desktop/clean/grounding/restart-device1erfolgreichZiel getroffen2.57 s
iot-control-desktop/clean/grounding/auto-mode1erfolgreichZiel getroffen2.08 s
iot-control-desktop/clean/grounding/pump-card1erfolgreichZiel getroffen2.26 s
iot-control-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering defects detected; elements are well-aligned, legible, and unclipped." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering defects detected; elements are well-aligned, legible, and unclipped." }12.60 s
iot-control-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The 'Maintenance scheduled' card overlaps the table content and separator lines." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The 'Maintenance scheduled' card overlaps the table content and separator lines." }4.28 s
iot-control-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.41 s
iot-control-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.22 s
banking-ios/clean/grounding/profile1erfolgreichZiel getroffen10.16 s
banking-ios/clean/grounding/transaction-search1erfolgreichZiel getroffen2.53 s
banking-ios/clean/grounding/hide-balance1erfolgreichZiel getroffen1.82 s
banking-ios/clean/grounding/transfer1erfolgreichZiel getroffen2.23 s
banking-ios/clean/layout-defect1erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 412, "y": 869, "width": 178, "height": 27 }, "description": "Two 'Transfer' text labels are vertically overlapping inside the button, creating a distorted, duplicated text rendering defect." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 412, "y": 869, "width": 178, "height": 27 }, "description": "Two 'Transfer' text labels are vertically overlapping inside the button, creating a distorted, duplicated text rendering defect." }13.11 s
banking-ios/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 59 }, "description": "The header text ('AYA' / 'banking') is cut off by the left viewport edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 59 }, "description": "The header text ('AYA' / 'banking') is cut off by the left viewport edge." }3.90 s
banking-ios/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.90 s
banking-ios/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.58 s
flight-pass-ios/clean/grounding/boarding-pass1erfolgreichZiel getroffen2.34 s
flight-pass-ios/clean/grounding/check-in1erfolgreichZiel getroffen2.16 s
flight-pass-ios/clean/grounding/trips-tab1erfolgreichZiel getroffen1.65 s
flight-pass-ios/clean/grounding/profile-tab1erfolgreichZiel getroffen1.71 s
flight-pass-ios/clean/layout-defect1erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 405, "y": 866, "width": 189, "height": 26 }, "description": "The \"Check in\" button text is clipped along its bottom edge inside the button container." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 405, "y": 866, "width": 189, "height": 26 }, "description": "The \"Check in\" button text is clipped along its bottom edge inside the button container." }12.77 s
flight-pass-ios/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 45, "y": 850, "width": 910, "height": 68 }, "description": "The red 'Flight update' banner overlaps and obscures the blue button beneath it." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 45, "y": 850, "width": 910, "height": 68 }, "description": "The red 'Flight update' banner overlaps and obscures the blue button beneath it." }4.16 s
flight-pass-ios/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.99 s
flight-pass-ios/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.89 s
smart-home-android/clean/grounding/more1erfolgreichZiel getroffen2.05 s
smart-home-android/clean/grounding/room-search1erfolgreichZiel getroffen2.36 s
smart-home-android/clean/grounding/thermostat-card1erfolgreichZiel getroffen2.22 s
smart-home-android/clean/grounding/add-device1erfolgreichZiel getroffen1.55 s
smart-home-android/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "None" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "None" }8.97 s
smart-home-android/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 355, "y": 510, "width": 600, "height": 69 }, "description": "The 'Kitchen lights' card is misaligned and unexpectedly indented/broken compared to the other list items in the 'Quick view' section." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 355, "y": 510, "width": 600, "height": 69 }, "description": "The 'Kitchen lights' card is misaligned and unexpectedly indented/broken compared to the other list items in the 'Quick view' section." }7.27 s
smart-home-android/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.35 s
smart-home-android/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.06 s
commerce-catalog-desktop/clean/grounding/search2erfolgreichZiel getroffen2.53 s
commerce-catalog-desktop/clean/grounding/add-product2erfolgreichZiel getroffen2.38 s
commerce-catalog-desktop/clean/grounding/catalog-query2erfolgreichZiel getroffen2.62 s
commerce-catalog-desktop/clean/grounding/stock-filter2erfolgreichZiel getroffen2.50 s
commerce-catalog-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects detected in the interface." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects detected in the interface." }4.11 s
commerce-catalog-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 68, "height": 73 }, "description": "The page header text ('Products' and category label above it) is horizontally clipped on the left edge by its container boundary." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 68, "height": 73 }, "description": "The page header text ('Products' and category label above it) is horizontally clipped on the left edge by its container boundary." }10.76 s
commerce-catalog-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.12 s
commerce-catalog-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }3.35 s
analytics-dashboard-desktop/clean/grounding/audience-tab2erfolgreichZiel getroffen8.75 s
analytics-dashboard-desktop/clean/grounding/revenue-card2erfolgreichZiel getroffen2.04 s
analytics-dashboard-desktop/clean/grounding/search-field2erfolgreichZiel getroffen1.87 s
analytics-dashboard-desktop/clean/grounding/compare-toggle2erfolgreichZiel getroffen2.24 s
analytics-dashboard-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering defects detected." }5.95 s
analytics-dashboard-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 224, "width": 339, "height": 142 }, "description": "The heading text inside the first KPI card is cut off and clipped by the card's right boundary." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 224, "width": 339, "height": 142 }, "description": "The heading text inside the first KPI card is cut off and clipped by the card's right boundary." }3.10 s
analytics-dashboard-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.67 s
analytics-dashboard-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.94 s
iot-control-desktop/clean/grounding/devices-nav2erfolgreichZiel getroffen2.15 s
iot-control-desktop/clean/grounding/restart-device2erfolgreichZiel getroffen2.22 s
iot-control-desktop/clean/grounding/auto-mode2erfolgreichZiel getroffen2.25 s
iot-control-desktop/clean/grounding/pump-card2erfolgreichZiel getroffen2.36 s
iot-control-desktop/clean/layout-defect2erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 185, "y": 118, "width": 227, "height": 70 }, "description": "The subtitle 'FACTORY FLOOR 02' overlaps directly onto the top of the main 'IoT control center' heading." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 185, "y": 118, "width": 227, "height": 70 }, "description": "The subtitle 'FACTORY FLOOR 02' overlaps directly onto the top of the main 'IoT control center' heading." }4.70 s
iot-control-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' badge or overlay is floating awkwardly and overlapping table row divider lines and table contents." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' badge or overlay is floating awkwardly and overlapping table row divider lines and table contents." }2.14 s
iot-control-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.26 s
iot-control-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.55 s
banking-ios/clean/grounding/profile2erfolgreichZiel getroffen5.36 s
banking-ios/clean/grounding/transaction-search2erfolgreichZiel getroffen2.49 s
banking-ios/clean/grounding/hide-balance2erfolgreichZiel getroffen2.10 s
banking-ios/clean/grounding/transfer2erfolgreichZiel getroffen2.07 s
banking-ios/clean/layout-defect2erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 413, "y": 868, "width": 174, "height": 22 }, "description": "The 'Transfer' button label has duplicated text overlapping vertically, rendering the text distorted and corrupted." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 413, "y": 868, "width": 174, "height": 22 }, "description": "The 'Transfer' button label has duplicated text overlapping vertically, rendering the text distorted and corrupted." }7.07 s
banking-ios/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 60 }, "description": "The header text in the upper left corner is clipped off by the left viewport edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 60 }, "description": "The header text in the upper left corner is clipped off by the left viewport edge." }4.05 s
banking-ios/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.54 s
banking-ios/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.07 s
flight-pass-ios/clean/grounding/boarding-pass2erfolgreichZiel getroffen2.06 s
flight-pass-ios/clean/grounding/check-in2erfolgreichZiel getroffen1.79 s
flight-pass-ios/clean/grounding/trips-tab2erfolgreichZiel getroffen2.85 s
flight-pass-ios/clean/grounding/profile-tab2erfolgreichZiel getroffen2.04 s
flight-pass-ios/clean/layout-defect2erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 852, "width": 908, "height": 59 }, "description": "The 'Check in' label inside the primary action button is vertically clipped at the bottom." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 852, "width": 908, "height": 59 }, "description": "The 'Check in' label inside the primary action button is vertically clipped at the bottom." }7.45 s
flight-pass-ios/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 46, "y": 849, "width": 908, "height": 67 }, "description": "The red 'Flight update' notification banner overlaps and obscures the blue button behind it." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 46, "y": 849, "width": 908, "height": 67 }, "description": "The red 'Flight update' notification banner overlaps and obscures the blue button behind it." }2.35 s
flight-pass-ios/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.73 s
flight-pass-ios/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.85 s
smart-home-android/clean/grounding/more2erfolgreichZiel getroffen2.12 s
smart-home-android/clean/grounding/room-search2erfolgreichZiel getroffen2.18 s
smart-home-android/clean/grounding/thermostat-card2erfolgreichZiel getroffen2.07 s
smart-home-android/clean/grounding/add-device2erfolgreichZiel getroffen2.02 s
smart-home-android/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No rendering defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No rendering defects detected." }6.38 s
smart-home-android/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 509, "width": 600, "height": 71 }, "description": "The second list card ('Kitchen lights') is severely misaligned horizontally compared to the other list items, breaking the expected vertical list layout." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 509, "width": 600, "height": 71 }, "description": "The second list card ('Kitchen lights') is severely misaligned horizontally compared to the other list items, breaking the expected vertical list layout." }8.71 s
smart-home-android/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.21 s
smart-home-android/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.38 s
commerce-catalog-desktop/clean/grounding/search3erfolgreichZiel getroffen2.32 s
commerce-catalog-desktop/clean/grounding/add-product3erfolgreichZiel getroffen2.40 s
commerce-catalog-desktop/clean/grounding/catalog-query3erfolgreichZiel getroffen2.62 s
commerce-catalog-desktop/clean/grounding/stock-filter3erfolgreichZiel getroffen3.04 s
commerce-catalog-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible defects detected." }10.99 s
commerce-catalog-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 127, "width": 68, "height": 67 }, "description": "The section title text ('ducts') and subtitle ('IANAGEMENT') are clipped on the left edge by the sidebar/container boundary." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 127, "width": 68, "height": 67 }, "description": "The section title text ('ducts') and subtitle ('IANAGEMENT') are clipped on the left edge by the sidebar/container boundary." }2.32 s
commerce-catalog-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.51 s
commerce-catalog-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.85 s
analytics-dashboard-desktop/clean/grounding/audience-tab3erfolgreichZiel getroffen9.88 s
analytics-dashboard-desktop/clean/grounding/revenue-card3erfolgreichZiel getroffen2.26 s
analytics-dashboard-desktop/clean/grounding/search-field3erfolgreichZiel getroffen2.06 s
analytics-dashboard-desktop/clean/grounding/compare-toggle3erfolgreichZiel getroffen2.07 s
analytics-dashboard-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual defects detected in the interface." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual defects detected in the interface." }5.59 s
analytics-dashboard-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 187, "y": 223, "width": 340, "height": 143 }, "description": "The metric card title 'Monthly revenue across all sales chann' is clipped by the container's right boundary." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 187, "y": 223, "width": 340, "height": 143 }, "description": "The metric card title 'Monthly revenue across all sales chann' is clipped by the container's right boundary." }5.54 s
analytics-dashboard-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.80 s
analytics-dashboard-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.89 s
iot-control-desktop/clean/grounding/devices-nav3erfolgreichZiel getroffen2.31 s
iot-control-desktop/clean/grounding/restart-device3erfolgreichZiel getroffen2.36 s
iot-control-desktop/clean/grounding/auto-mode3erfolgreichZiel getroffen2.29 s
iot-control-desktop/clean/grounding/pump-card3erfolgreichZiel getroffen2.74 s
iot-control-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects detected in the interface." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects detected in the interface." }11.74 s
iot-control-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 223, "height": 67 }, "description": "The 'Maintenance scheduled' banner overlaps directly on top of the data table and its row borders." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 223, "height": 67 }, "description": "The 'Maintenance scheduled' banner overlaps directly on top of the data table and its row borders." }13.83 s
iot-control-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.37 s
iot-control-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.58 s
banking-ios/clean/grounding/profile3erfolgreichZiel getroffen12.22 s
banking-ios/clean/grounding/transaction-search3erfolgreichZiel getroffen3.27 s
banking-ios/clean/grounding/hide-balance3erfolgreichZiel getroffen2.09 s
banking-ios/clean/grounding/transfer3erfolgreichZiel getroffen1.99 s
banking-ios/clean/layout-defect3erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 410, "y": 866, "width": 180, "height": 25 }, "description": "Two text labels within the 'Transfer' button overlap directly on top of each other, creating garbled and obscured text." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 410, "y": 866, "width": 180, "height": 25 }, "description": "Two text labels within the 'Transfer' button overlap directly on top of each other, creating garbled and obscured text." }8.34 s
banking-ios/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 62 }, "description": "The title text 'banking' and the header label above it are cropped by the left edge of the viewport." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 62 }, "description": "The title text 'banking' and the header label above it are cropped by the left edge of the viewport." }3.69 s
banking-ios/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.23 s
banking-ios/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.50 s
flight-pass-ios/clean/grounding/boarding-pass3erfolgreichZiel getroffen2.05 s
flight-pass-ios/clean/grounding/check-in3erfolgreichZiel getroffen2.25 s
flight-pass-ios/clean/grounding/trips-tab3erfolgreichZiel getroffen2.51 s
flight-pass-ios/clean/grounding/profile-tab3erfolgreichZiel getroffen1.92 s
flight-pass-ios/clean/layout-defect3ungültige AntwortThe answer is not valid JSON. Parsing failed before schema validation could run. JSON parsing error: Unterminated string in JSON at position 213 (line 1 column 214) Malformed answers are not repaired in this pilot and count as incorrect, even when the intended defect category is recognizable. Expected: { "has_issue": false, "category": null, "region": null } Returned: {"has_issue": true, "category": "container_clipping", "region": {"x": 409, "y": 869, "width": 182, "height": 22}, "description": "The bottom portion of the 'Check in' button text is clipped horizontally inside its · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: {"has_issue": true, "category": "container_clipping", "region": {"x": 409, "y": 869, "width": 182, "height": 22}, "description": "The bottom portion of the 'Check in' button text is clipped horizontally inside its17.47 s
flight-pass-ios/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 66 }, "description": "A red 'Flight update' notification banner overlaps and obscures the blue 'Check in' action button." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 66 }, "description": "A red 'Flight update' notification banner overlaps and obscures the blue 'Check in' action button." }4.31 s
flight-pass-ios/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.41 s
flight-pass-ios/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.49 s
smart-home-android/clean/grounding/more3erfolgreichZiel getroffen1.79 s
smart-home-android/clean/grounding/room-search3erfolgreichZiel getroffen2.07 s
smart-home-android/clean/grounding/thermostat-card3erfolgreichZiel getroffen2.30 s
smart-home-android/clean/grounding/add-device3erfolgreichZiel getroffen2.25 s
smart-home-android/clean/layout-defect3erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 140, "y": 966, "width": 725, "height": 34 }, "description": "The bottom navigation bar labels ('Home', 'Rooms', 'Activity') are partially cut off by the bottom viewport edge." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 140, "y": 966, "width": 725, "height": 34 }, "description": "The bottom navigation bar labels ('Home', 'Rooms', 'Activity') are partially cut off by the bottom viewport edge." }8.35 s
smart-home-android/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 355, "y": 510, "width": 600, "height": 70 }, "description": "The 'Kitchen lights' list item card is significantly shifted and misaligned from the vertical grid layout used by surrounding items." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 355, "y": 510, "width": 600, "height": 70 }, "description": "The 'Kitchen lights' list item card is significantly shifted and misaligned from the vertical grid layout used by surrounding items." }8.83 s
smart-home-android/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.84 s
smart-home-android/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.62 s
Grok 4.7 · 144 Fallausführungen

Zurückgegebene Modellversionen: x-ai/grok-4.7.

Ziele mit demselben Treffer-/Fehlerergebnis in allen drei Wiederholungen: 16 / 24. Übereinstimmung kann auch bedeuten, dass ein Ziel jedes Mal verfehlt wird.

FallWiederholungErgebnisAusgangZeit
commerce-catalog-desktop/clean/grounding/search1erfolgreichZiel getroffen12.82 s
commerce-catalog-desktop/clean/grounding/add-product1erfolgreichZiel getroffen7.11 s
commerce-catalog-desktop/clean/grounding/catalog-query1erfolgreichZiel getroffen9.00 s
commerce-catalog-desktop/clean/grounding/stock-filter1erfolgreichZiel getroffen28.50 s
commerce-catalog-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect." }12.36 s
commerce-catalog-desktop/defect/layout-defect1ungültige AntwortThe answer is not valid JSON. Parsing failed before schema validation could run. JSON parsing error: Unterminated string in JSON at position 195 (line 1 column 196) Malformed answers are not repaired in this pilot and count as incorrect, even when the intended defect category is recognizable. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: {"has_issue":true,"category":"container_clipping","region":{"x":278,"y":114,"width":99,"height":65},"description":"The page heading is clipped on the left, showing only “IANAGEMENT” and “ducts.”} · The answer is not valid JSON. Parsing failed before schema validation could run. JSON parsing error: Unterminated string in JSON at position 195 (line 1 column 196) Malformed answers are not repaired in this pilot and count as incorrect, even when the intended defect category is recognizable. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: {"has_issue":true,"category":"container_clipping","region":{"x":278,"y":114,"width":99,"height":65},"description":"The page heading is clipped on the left, showing only “IANAGEMENT” and “ducts.”}3.06 s
commerce-catalog-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.39 s
commerce-catalog-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.97 s
analytics-dashboard-desktop/clean/grounding/audience-tab1erfolgreichZiel getroffen26.90 s
analytics-dashboard-desktop/clean/grounding/revenue-card1erfolgreichZiel getroffen9.62 s
analytics-dashboard-desktop/clean/grounding/search-field1erfolgreichZiel getroffen11.49 s
analytics-dashboard-desktop/clean/grounding/compare-toggle1erfolgreichZiel verfehlt27.51 s
analytics-dashboard-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, overflow, overlap, obscured control, or broken reflow is visible." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, overflow, overlap, obscured control, or broken reflow is visible." }92.19 s
analytics-dashboard-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 204, "y": 226, "width": 318, "height": 34 }, "description": "The revenue card title is truncated mid-word at the card's right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 204, "y": 226, "width": 318, "height": 34 }, "description": "The revenue card title is truncated mid-word at the card's right edge." }18.34 s
analytics-dashboard-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.54 s
analytics-dashboard-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }18.39 s
iot-control-desktop/clean/grounding/devices-nav1erfolgreichZiel getroffen13.27 s
iot-control-desktop/clean/grounding/restart-device1erfolgreichZiel getroffen15.28 s
iot-control-desktop/clean/grounding/auto-mode1erfolgreichZiel getroffen25.79 s
iot-control-desktop/clean/grounding/pump-card1erfolgreichZiel getroffen14.44 s
iot-control-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect; all text, controls, cards, and table rows are fully contained and aligned." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect; all text, controls, cards, and table rows are fully contained and aligned." }6.73 s
iot-control-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 556, "width": 219, "height": 66 }, "description": "A red 'Maintenance scheduled' banner overlaps the device table rows." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 556, "width": 219, "height": 66 }, "description": "A red 'Maintenance scheduled' banner overlaps the device table rows." }25.49 s
iot-control-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.88 s
iot-control-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.87 s
banking-ios/clean/grounding/profile1erfolgreichZiel verfehlt21.93 s
banking-ios/clean/grounding/transaction-search1erfolgreichZiel getroffen10.19 s
banking-ios/clean/grounding/hide-balance1erfolgreichZiel getroffen54.00 s
banking-ios/clean/grounding/transfer1erfolgreichZiel getroffen11.31 s
banking-ios/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the listed categories." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the listed categories." }8.19 s
banking-ios/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 224, "height": 34 }, "description": "The 'Banking' heading is cropped by the left viewport edge, leaving only 'anking' visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 224, "height": 34 }, "description": "The 'Banking' heading is cropped by the left viewport edge, leaving only 'anking' visible." }20.63 s
banking-ios/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.66 s
banking-ios/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.00 s
flight-pass-ios/clean/grounding/boarding-pass1erfolgreichZiel getroffen14.68 s
flight-pass-ios/clean/grounding/check-in1erfolgreichZiel getroffen8.91 s
flight-pass-ios/clean/grounding/trips-tab1erfolgreichZiel getroffen17.82 s
flight-pass-ios/clean/grounding/profile-tab1erfolgreichZiel getroffen17.09 s
flight-pass-ios/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }12.00 s
flight-pass-ios/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 48, "y": 842, "width": 904, "height": 90 }, "description": "Red Flight update banner overlaps a blue banner, obscuring its text." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 48, "y": 842, "width": 904, "height": 90 }, "description": "Red Flight update banner overlaps a blue banner, obscuring its text." }19.37 s
flight-pass-ios/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.86 s
flight-pass-ios/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.58 s
smart-home-android/clean/grounding/more1erfolgreichZiel verfehlt8.75 s
smart-home-android/clean/grounding/room-search1erfolgreichZiel getroffen9.33 s
smart-home-android/clean/grounding/thermostat-card1erfolgreichZiel getroffen13.72 s
smart-home-android/clean/grounding/add-device1erfolgreichZiel verfehlt8.84 s
smart-home-android/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }3.17 s
smart-home-android/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 366, "y": 509, "width": 588, "height": 83 }, "description": "The Kitchen lights card is shifted right and does not align with the full-width cards above and below it." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 366, "y": 509, "width": 588, "height": 83 }, "description": "The Kitchen lights card is shifted right and does not align with the full-width cards above and below it." }20.28 s
smart-home-android/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.88 s
smart-home-android/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.52 s
commerce-catalog-desktop/clean/grounding/search2erfolgreichZiel getroffen26.80 s
commerce-catalog-desktop/clean/grounding/add-product2erfolgreichZiel getroffen7.36 s
commerce-catalog-desktop/clean/grounding/catalog-query2erfolgreichZiel getroffen20.98 s
commerce-catalog-desktop/clean/grounding/stock-filter2erfolgreichZiel verfehlt17.11 s
commerce-catalog-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }5.63 s
commerce-catalog-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 276, "y": 112, "width": 103, "height": 68 }, "description": "The page heading is clipped at its left edge, cutting off the beginning of both heading lines." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 276, "y": 112, "width": 103, "height": 68 }, "description": "The page heading is clipped at its left edge, cutting off the beginning of both heading lines." }3.07 s
commerce-catalog-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.73 s
commerce-catalog-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.08 s
analytics-dashboard-desktop/clean/grounding/audience-tab2erfolgreichZiel getroffen18.38 s
analytics-dashboard-desktop/clean/grounding/revenue-card2erfolgreichZiel getroffen16.61 s
analytics-dashboard-desktop/clean/grounding/search-field2erfolgreichZiel verfehlt38.77 s
analytics-dashboard-desktop/clean/grounding/compare-toggle2erfolgreichZiel verfehlt21.24 s
analytics-dashboard-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect." }45.43 s
analytics-dashboard-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 228, "width": 324, "height": 46 }, "description": "The metric card heading is cut off mid-word at the card's right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 228, "width": 324, "height": 46 }, "description": "The metric card heading is cut off mid-word at the card's right edge." }14.96 s
analytics-dashboard-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.22 s
analytics-dashboard-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.97 s
iot-control-desktop/clean/grounding/devices-nav2erfolgreichZiel getroffen21.10 s
iot-control-desktop/clean/grounding/restart-device2erfolgreichZiel getroffen11.53 s
iot-control-desktop/clean/grounding/auto-mode2erfolgreichZiel verfehlt16.62 s
iot-control-desktop/clean/grounding/pump-card2erfolgreichZiel getroffen22.31 s
iot-control-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects." }13.76 s
iot-control-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 458, "y": 555, "width": 224, "height": 69 }, "description": "A red \"Maintenance scheduled\" banner overlaps and obscures the device table rows." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 458, "y": 555, "width": 224, "height": 69 }, "description": "A red \"Maintenance scheduled\" banner overlaps and obscures the device table rows." }12.82 s
iot-control-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.78 s
iot-control-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.89 s
banking-ios/clean/grounding/profile2erfolgreichZiel verfehlt20.19 s
banking-ios/clean/grounding/transaction-search2erfolgreichZiel getroffen24.96 s
banking-ios/clean/grounding/hide-balance2erfolgreichZiel verfehlt8.56 s
banking-ios/clean/grounding/transfer2erfolgreichZiel getroffen8.18 s
banking-ios/clean/layout-defect2erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 58, "y": 962, "width": 814, "height": 38 }, "description": "Bottom navigation labels are cut off by the lower viewport edge." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 58, "y": 962, "width": 814, "height": 38 }, "description": "Bottom navigation labels are cut off by the lower viewport edge." }18.85 s
banking-ios/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 58, "width": 500, "height": 48 }, "description": "The 'Banking' heading is cropped at the left viewport edge, with the leading 'B' cut off." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 58, "width": 500, "height": 48 }, "description": "The 'Banking' heading is cropped at the left viewport edge, with the leading 'B' cut off." }20.32 s
banking-ios/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.54 s
banking-ios/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.83 s
flight-pass-ios/clean/grounding/boarding-pass2erfolgreichZiel getroffen12.05 s
flight-pass-ios/clean/grounding/check-in2erfolgreichZiel verfehlt15.24 s
flight-pass-ios/clean/grounding/trips-tab2erfolgreichZiel getroffen23.46 s
flight-pass-ios/clean/grounding/profile-tab2erfolgreichZiel getroffen25.79 s
flight-pass-ios/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect; content fits the viewport without clipping, overflow, overlap, or broken reflow." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect; content fits the viewport without clipping, overflow, overlap, or broken reflow." }7.64 s
flight-pass-ios/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 42, "y": 812, "width": 918, "height": 92 }, "description": "A red flight-update banner overlaps a blue card behind it, obscuring that card's text." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 42, "y": 812, "width": 918, "height": 92 }, "description": "A red flight-update banner overlaps a blue card behind it, obscuring that card's text." }12.45 s
flight-pass-ios/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.96 s
flight-pass-ios/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.77 s
smart-home-android/clean/grounding/more2erfolgreichZiel verfehlt16.25 s
smart-home-android/clean/grounding/room-search2erfolgreichZiel getroffen16.47 s
smart-home-android/clean/grounding/thermostat-card2erfolgreichZiel getroffen19.16 s
smart-home-android/clean/grounding/add-device2erfolgreichZiel getroffen9.97 s
smart-home-android/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect." }15.40 s
smart-home-android/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 328, "y": 567, "width": 632, "height": 110 }, "description": "The Kitchen lights card is shifted right and overlaps the Front door card beneath it." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 328, "y": 567, "width": 632, "height": 110 }, "description": "The Kitchen lights card is shifted right and overlaps the Front door card beneath it." }40.90 s
smart-home-android/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.51 s
smart-home-android/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }5.93 s
commerce-catalog-desktop/clean/grounding/search3erfolgreichZiel getroffen27.72 s
commerce-catalog-desktop/clean/grounding/add-product3ungültige AntwortZiel verfehlt11.21 s
commerce-catalog-desktop/clean/grounding/catalog-query3erfolgreichZiel getroffen21.57 s
commerce-catalog-desktop/clean/grounding/stock-filter3erfolgreichZiel verfehlt17.41 s
commerce-catalog-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, overflow, overlap, obscured control, or broken reflow is visible." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, overflow, overlap, obscured control, or broken reflow is visible." }13.76 s
commerce-catalog-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 273, "y": 116, "width": 102, "height": 62 }, "description": "Heading text is clipped at its left edge, showing incomplete words." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 273, "y": 116, "width": 102, "height": 62 }, "description": "Heading text is clipped at its left edge, showing incomplete words." }3.93 s
commerce-catalog-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.19 s
commerce-catalog-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.49 s
analytics-dashboard-desktop/clean/grounding/audience-tab3erfolgreichZiel getroffen24.18 s
analytics-dashboard-desktop/clean/grounding/revenue-card3erfolgreichZiel getroffen9.67 s
analytics-dashboard-desktop/clean/grounding/search-field3erfolgreichZiel getroffen6.95 s
analytics-dashboard-desktop/clean/grounding/compare-toggle3erfolgreichZiel verfehlt10.42 s
analytics-dashboard-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the listed categories." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the listed categories." }35.63 s
analytics-dashboard-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 294, "y": 224, "width": 459, "height": 34 }, "description": "The revenue card heading is cut off at the card's right edge, truncating the title text." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 294, "y": 224, "width": 459, "height": 34 }, "description": "The revenue card heading is cut off at the card's right edge, truncating the title text." }18.82 s
analytics-dashboard-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }6.03 s
analytics-dashboard-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.56 s
iot-control-desktop/clean/grounding/devices-nav3erfolgreichZiel getroffen13.95 s
iot-control-desktop/clean/grounding/restart-device3erfolgreichZiel getroffen8.91 s
iot-control-desktop/clean/grounding/auto-mode3erfolgreichZiel getroffen18.87 s
iot-control-desktop/clean/grounding/pump-card3erfolgreichZiel getroffen19.79 s
iot-control-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect; text, cards, table, and controls render within their containers." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect; text, cards, table, and controls render within their containers." }13.84 s
iot-control-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 463, "y": 556, "width": 223, "height": 68 }, "description": "A red maintenance banner overlaps the device table, covering the Cooling pump status and row boundary." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 463, "y": 556, "width": 223, "height": 68 }, "description": "A red maintenance banner overlaps the device table, covering the Cooling pump status and row boundary." }5.51 s
iot-control-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.98 s
iot-control-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.08 s
banking-ios/clean/grounding/profile3erfolgreichZiel verfehlt17.73 s
banking-ios/clean/grounding/transaction-search3erfolgreichZiel getroffen17.27 s
banking-ios/clean/grounding/hide-balance3erfolgreichZiel verfehlt21.90 s
banking-ios/clean/grounding/transfer3erfolgreichZiel getroffen7.91 s
banking-ios/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }9.43 s
banking-ios/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 49, "width": 162, "height": 44 }, "description": "The Banking header title is cropped at the left viewport edge, leaving only anking visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 49, "width": 162, "height": 44 }, "description": "The Banking header title is cropped at the left viewport edge, leaving only anking visible." }12.40 s
banking-ios/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.41 s
banking-ios/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.02 s
flight-pass-ios/clean/grounding/boarding-pass3erfolgreichZiel getroffen12.12 s
flight-pass-ios/clean/grounding/check-in3erfolgreichZiel verfehlt17.61 s
flight-pass-ios/clean/grounding/trips-tab3erfolgreichZiel getroffen18.95 s
flight-pass-ios/clean/grounding/profile-tab3erfolgreichZiel getroffen32.81 s
flight-pass-ios/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect in the specified categories." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect in the specified categories." }10.62 s
flight-pass-ios/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 41, "y": 829, "width": 877, "height": 90 }, "description": "The red Flight update banner overlaps the blue banner behind it, obscuring and clipping its text." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 41, "y": 829, "width": 877, "height": 90 }, "description": "The red Flight update banner overlaps the blue banner behind it, obscuring and clipping its text." }17.93 s
flight-pass-ios/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.46 s
flight-pass-ios/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.66 s
smart-home-android/clean/grounding/more3erfolgreichZiel verfehlt4.33 s
smart-home-android/clean/grounding/room-search3erfolgreichZiel getroffen6.48 s
smart-home-android/clean/grounding/thermostat-card3erfolgreichZiel verfehlt22.65 s
smart-home-android/clean/grounding/add-device3erfolgreichZiel getroffen9.77 s
smart-home-android/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the specified categories." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the specified categories." }10.69 s
smart-home-android/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 430, "y": 510, "width": 528, "height": 80 }, "description": "The Kitchen lights card is indented and narrower than the full-width sibling cards, breaking the list reflow." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 430, "y": 510, "width": 528, "height": 80 }, "description": "The Kitchen lights card is indented and narrower than the full-width sibling cards, breaking the list reflow." }23.64 s
smart-home-android/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.57 s
smart-home-android/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.55 s
Qwen3.8 Max (0902) · 144 Fallausführungen

Zurückgegebene Modellversionen: qwen/qwen3.8-max-0902.

Ziele mit demselben Treffer-/Fehlerergebnis in allen drei Wiederholungen: 24 / 24. Übereinstimmung kann auch bedeuten, dass ein Ziel jedes Mal verfehlt wird.

FallWiederholungErgebnisAusgangZeit
commerce-catalog-desktop/clean/grounding/search1erfolgreichZiel getroffen6.70 s
commerce-catalog-desktop/clean/grounding/add-product1erfolgreichZiel getroffen9.03 s
commerce-catalog-desktop/clean/grounding/catalog-query1erfolgreichZiel getroffen9.96 s
commerce-catalog-desktop/clean/grounding/stock-filter1erfolgreichZiel getroffen7.91 s
commerce-catalog-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." }11.01 s
commerce-catalog-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 190, "y": 120, "width": 180, "height": 75 }, "description": "Page heading is cut off at the left edge, showing 'ducts' instead of 'Products' and 'IANAGEMENT' instead of 'MANAGEMENT'." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 190, "y": 120, "width": 180, "height": 75 }, "description": "Page heading is cut off at the left edge, showing 'ducts' instead of 'Products' and 'IANAGEMENT' instead of 'MANAGEMENT'." }15.27 s
commerce-catalog-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }4.24 s
commerce-catalog-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }5.16 s
analytics-dashboard-desktop/clean/grounding/audience-tab1erfolgreichZiel getroffen14.38 s
analytics-dashboard-desktop/clean/grounding/revenue-card1erfolgreichZiel getroffen11.24 s
analytics-dashboard-desktop/clean/grounding/search-field1erfolgreichZiel getroffen10.90 s
analytics-dashboard-desktop/clean/grounding/compare-toggle1erfolgreichZiel getroffen11.42 s
analytics-dashboard-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." }11.64 s
analytics-dashboard-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 200, "y": 245, "width": 330, "height": 40 }, "description": "Card title 'Monthly revenue across all sales channels' is cut off at the card's right edge, truncating the word 'channels'." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 200, "y": 245, "width": 330, "height": 40 }, "description": "Card title 'Monthly revenue across all sales channels' is cut off at the card's right edge, truncating the word 'channels'." }14.56 s
analytics-dashboard-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }4.22 s
analytics-dashboard-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }5.52 s
iot-control-desktop/clean/grounding/devices-nav1erfolgreichZiel getroffen11.78 s
iot-control-desktop/clean/grounding/restart-device1erfolgreichZiel getroffen11.50 s
iot-control-desktop/clean/grounding/auto-mode1erfolgreichZiel getroffen14.37 s
iot-control-desktop/clean/grounding/pump-card1erfolgreichZiel getroffen8.43 s
iot-control-desktop/clean/layout-defect1ungültige AntwortThe answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: cards, table, toggle, and button render fully within their containers without cropping, clipping, overflow, overlap, or obscured controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: cards, table, toggle, and button render fully within their containers without cropping, clipping, overflow, overlap, or obscured controls." }10.94 s
iot-control-desktop/defect/layout-defect1erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; toast notification overlays content intentionally without obscuring actionable controls." } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; toast notification overlays content intentionally without obscuring actionable controls." }14.07 s
iot-control-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.83 s
iot-control-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }6.18 s
banking-ios/clean/grounding/profile1erfolgreichZiel getroffen9.52 s
banking-ios/clean/grounding/transaction-search1erfolgreichZiel getroffen6.30 s
banking-ios/clean/grounding/hide-balance1erfolgreichZiel getroffen18.69 s
banking-ios/clean/grounding/transfer1erfolgreichZiel getroffen11.96 s
banking-ios/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all containers, text, and controls render fully within the viewport without clipping, overflow, or overlap." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all containers, text, and controls render fully within the viewport without clipping, overflow, or overlap." }9.44 s
banking-ios/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 280, "height": 60 }, "description": "Page heading 'Banking' is clipped at the left edge; the leading 'B' is cut off by the viewport." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 280, "height": 60 }, "description": "Page heading 'Banking' is clipped at the left edge; the leading 'B' is cut off by the viewport." }9.56 s
banking-ios/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.62 s
banking-ios/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }4.11 s
flight-pass-ios/clean/grounding/boarding-pass1erfolgreichZiel getroffen8.08 s
flight-pass-ios/clean/grounding/check-in1erfolgreichZiel getroffen6.42 s
flight-pass-ios/clean/grounding/trips-tab1erfolgreichZiel getroffen8.58 s
flight-pass-ios/clean/grounding/profile-tab1erfolgreichZiel getroffen7.65 s
flight-pass-ios/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." }8.78 s
flight-pass-ios/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 855, "width": 920, "height": 65 }, "description": "Red 'Flight update' toast banner overlaps and obscures the blue 'Check in' button text beneath it." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 855, "width": 920, "height": 65 }, "description": "Red 'Flight update' toast banner overlaps and obscures the blue 'Check in' button text beneath it." }14.78 s
flight-pass-ios/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.73 s
flight-pass-ios/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }4.51 s
smart-home-android/clean/grounding/more1erfolgreichZiel getroffen9.39 s
smart-home-android/clean/grounding/room-search1erfolgreichZiel getroffen5.70 s
smart-home-android/clean/grounding/thermostat-card1erfolgreichZiel getroffen7.16 s
smart-home-android/clean/grounding/add-device1erfolgreichZiel getroffen6.43 s
smart-home-android/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without overlap or clipping." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without overlap or clipping." }9.54 s
smart-home-android/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 355, "y": 510, "width": 645, "height": 70 }, "description": "The 'Kitchen lights' card (item 2) is shifted right and clipped at the right viewport edge, cutting off its border and chevron control." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 355, "y": 510, "width": 645, "height": 70 }, "description": "The 'Kitchen lights' card (item 2) is shifted right and clipped at the right viewport edge, cutting off its border and chevron control." }14.85 s
smart-home-android/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.91 s
smart-home-android/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }4.46 s
commerce-catalog-desktop/clean/grounding/search2erfolgreichZiel getroffen7.87 s
commerce-catalog-desktop/clean/grounding/add-product2erfolgreichZiel getroffen10.41 s
commerce-catalog-desktop/clean/grounding/catalog-query2erfolgreichZiel getroffen10.06 s
commerce-catalog-desktop/clean/grounding/stock-filter2erfolgreichZiel getroffen12.58 s
commerce-catalog-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all content fits within containers and viewport." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all content fits within containers and viewport." }8.64 s
commerce-catalog-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 190, "y": 120, "width": 180, "height": 75 }, "description": "Page heading 'Products' is clipped on the left, showing 'ducts', and the 'MANAGEMENT' eyebrow label is cut off at the container edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 190, "y": 120, "width": 180, "height": 75 }, "description": "Page heading 'Products' is clipped on the left, showing 'ducts', and the 'MANAGEMENT' eyebrow label is cut off at the container edge." }13.24 s
commerce-catalog-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.67 s
commerce-catalog-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }5.86 s
analytics-dashboard-desktop/clean/grounding/audience-tab2erfolgreichZiel getroffen11.26 s
analytics-dashboard-desktop/clean/grounding/revenue-card2erfolgreichZiel getroffen9.96 s
analytics-dashboard-desktop/clean/grounding/search-field2erfolgreichZiel getroffen9.24 s
analytics-dashboard-desktop/clean/grounding/compare-toggle2erfolgreichZiel getroffen12.36 s
analytics-dashboard-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all content fits within containers and viewport." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all content fits within containers and viewport." }10.56 s
analytics-dashboard-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 245, "width": 340, "height": 40 }, "description": "Card title 'Monthly revenue across all sales channels' is truncated at the right edge of its container, cutting off the final letters." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 245, "width": 340, "height": 40 }, "description": "Card title 'Monthly revenue across all sales channels' is truncated at the right edge of its container, cutting off the final letters." }13.44 s
analytics-dashboard-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }4.18 s
analytics-dashboard-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }5.57 s
iot-control-desktop/clean/grounding/devices-nav2erfolgreichZiel getroffen6.57 s
iot-control-desktop/clean/grounding/restart-device2erfolgreichZiel getroffen8.10 s
iot-control-desktop/clean/grounding/auto-mode2erfolgreichZiel getroffen10.17 s
iot-control-desktop/clean/grounding/pump-card2erfolgreichZiel getroffen5.96 s
iot-control-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without overlap or clipping." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without overlap or clipping." }9.94 s
iot-control-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 223, "height": 63 }, "description": "A red 'Maintenance scheduled' toast/banner overlays the device table rows, covering the area between the Cooling pump and Intake valve entries." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 223, "height": 63 }, "description": "A red 'Maintenance scheduled' toast/banner overlays the device table rows, covering the area between the Cooling pump and Intake valve entries." }22.10 s
iot-control-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.75 s
iot-control-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }5.03 s
banking-ios/clean/grounding/profile2erfolgreichZiel getroffen8.60 s
banking-ios/clean/grounding/transaction-search2erfolgreichZiel getroffen8.13 s
banking-ios/clean/grounding/hide-balance2erfolgreichZiel getroffen6.74 s
banking-ios/clean/grounding/transfer2erfolgreichZiel getroffen6.15 s
banking-ios/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." }8.60 s
banking-ios/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 300, "height": 60 }, "description": "Page title 'Banking' is cut off at the left viewport edge, showing only 'anking'." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 300, "height": 60 }, "description": "Page title 'Banking' is cut off at the left viewport edge, showing only 'anking'." }11.66 s
banking-ios/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.97 s
banking-ios/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }6.69 s
flight-pass-ios/clean/grounding/boarding-pass2erfolgreichZiel getroffen10.94 s
flight-pass-ios/clean/grounding/check-in2erfolgreichZiel getroffen6.76 s
flight-pass-ios/clean/grounding/trips-tab2erfolgreichZiel getroffen7.43 s
flight-pass-ios/clean/grounding/profile-tab2erfolgreichZiel getroffen7.58 s
flight-pass-ios/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all text fits, controls are unobscured, and elements are properly spaced." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all text fits, controls are unobscured, and elements are properly spaced." }9.35 s
flight-pass-ios/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 920, "height": 70 }, "description": "The red 'Flight update' toast banner overlaps and obscures the blue 'Check in' button text beneath it." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 920, "height": 70 }, "description": "The red 'Flight update' toast banner overlaps and obscures the blue 'Check in' button text beneath it." }11.78 s
flight-pass-ios/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.27 s
flight-pass-ios/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }5.39 s
smart-home-android/clean/grounding/more2erfolgreichZiel getroffen6.06 s
smart-home-android/clean/grounding/room-search2erfolgreichZiel getroffen6.43 s
smart-home-android/clean/grounding/thermostat-card2erfolgreichZiel getroffen6.58 s
smart-home-android/clean/grounding/add-device2erfolgreichZiel getroffen6.15 s
smart-home-android/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without cropping, clipping, overflow, overlap, or obscured controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without cropping, clipping, overflow, overlap, or obscured controls." }9.39 s
smart-home-android/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 350, "y": 510, "width": 650, "height": 75 }, "description": "The 'Kitchen lights' quick-view card is offset to the right and clipped by the right viewport edge, breaking the list's alignment." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 350, "y": 510, "width": 650, "height": 75 }, "description": "The 'Kitchen lights' quick-view card is offset to the right and clipped by the right viewport edge, breaking the list's alignment." }17.60 s
smart-home-android/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }4.70 s
smart-home-android/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }6.14 s
commerce-catalog-desktop/clean/grounding/search3erfolgreichZiel getroffen9.29 s
commerce-catalog-desktop/clean/grounding/add-product3erfolgreichZiel getroffen10.12 s
commerce-catalog-desktop/clean/grounding/catalog-query3erfolgreichZiel getroffen9.57 s
commerce-catalog-desktop/clean/grounding/stock-filter3erfolgreichZiel getroffen5.48 s
commerce-catalog-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within their containers and the viewport." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within their containers and the viewport." }7.69 s
commerce-catalog-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 165, "y": 110, "width": 120, "height": 70 }, "description": "Page heading is cut off on the left: 'MANAGEMENT' appears as 'IANAGEMENT' and 'Products' as 'ducts', clipped by the sidebar edge." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 165, "y": 110, "width": 120, "height": 70 }, "description": "Page heading is cut off on the left: 'MANAGEMENT' appears as 'IANAGEMENT' and 'Products' as 'ducts', clipped by the sidebar edge." }16.10 s
commerce-catalog-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }4.78 s
commerce-catalog-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }8.14 s
analytics-dashboard-desktop/clean/grounding/audience-tab3erfolgreichZiel getroffen7.93 s
analytics-dashboard-desktop/clean/grounding/revenue-card3erfolgreichZiel getroffen12.24 s
analytics-dashboard-desktop/clean/grounding/search-field3erfolgreichZiel getroffen15.80 s
analytics-dashboard-desktop/clean/grounding/compare-toggle3erfolgreichZiel getroffen11.01 s
analytics-dashboard-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." }10.27 s
analytics-dashboard-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 248, "width": 328, "height": 32 }, "description": "Card title \"Monthly revenue across all sales chann\" is truncated at the card's right edge, cutting off the word \"channels\"." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 248, "width": 328, "height": 32 }, "description": "Card title \"Monthly revenue across all sales chann\" is truncated at the card's right edge, cutting off the word \"channels\"." }12.46 s
analytics-dashboard-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }7.10 s
analytics-dashboard-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }4.95 s
iot-control-desktop/clean/grounding/devices-nav3erfolgreichZiel getroffen5.90 s
iot-control-desktop/clean/grounding/restart-device3erfolgreichZiel getroffen8.96 s
iot-control-desktop/clean/grounding/auto-mode3erfolgreichZiel getroffen7.81 s
iot-control-desktop/clean/grounding/pump-card3erfolgreichZiel getroffen8.01 s
iot-control-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects; layout is intact with no cropping, clipping, overflow, overlap, or obscured controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects; layout is intact with no cropping, clipping, overflow, overlap, or obscured controls." }10.18 s
iot-control-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 222, "height": 65 }, "description": "A red 'Maintenance scheduled' toast/banner overlaps the device table rows, covering content between the Cooling pump and Intake valve rows." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 222, "height": 65 }, "description": "A red 'Maintenance scheduled' toast/banner overlaps the device table rows, covering content between the Cooling pump and Intake valve rows." }11.95 s
iot-control-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.77 s
iot-control-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }6.03 s
banking-ios/clean/grounding/profile3erfolgreichZiel getroffen9.36 s
banking-ios/clean/grounding/transaction-search3erfolgreichZiel getroffen6.13 s
banking-ios/clean/grounding/hide-balance3erfolgreichZiel getroffen8.05 s
banking-ios/clean/grounding/transfer3erfolgreichZiel getroffen6.87 s
banking-ios/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without cropping, clipping, overflow, overlap, or obscured controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without cropping, clipping, overflow, overlap, or obscured controls." }9.48 s
banking-ios/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 280, "height": 50 }, "description": "Page title 'Banking' is cut off at the left viewport edge, showing only 'anking'." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 280, "height": 50 }, "description": "Page title 'Banking' is cut off at the left viewport edge, showing only 'anking'." }9.28 s
banking-ios/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }4.25 s
banking-ios/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }4.52 s
flight-pass-ios/clean/grounding/boarding-pass3erfolgreichZiel getroffen7.59 s
flight-pass-ios/clean/grounding/check-in3erfolgreichZiel getroffen9.32 s
flight-pass-ios/clean/grounding/trips-tab3erfolgreichZiel getroffen6.66 s
flight-pass-ios/clean/grounding/profile-tab3erfolgreichZiel getroffen8.68 s
flight-pass-ios/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all text fits within containers, controls are unobscured, and elements do not overlap or clip." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all text fits within containers, controls are unobscured, and elements do not overlap or clip." }8.34 s
flight-pass-ios/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 45, "y": 850, "width": 910, "height": 70 }, "description": "A red 'Flight update' toast banner overlaps and obscures a blue action button ('Check in') near the bottom of the screen." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 45, "y": 850, "width": 910, "height": 70 }, "description": "A red 'Flight update' toast banner overlaps and obscures a blue action button ('Check in') near the bottom of the screen." }14.65 s
flight-pass-ios/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.98 s
flight-pass-ios/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }4.74 s
smart-home-android/clean/grounding/more3erfolgreichZiel getroffen8.92 s
smart-home-android/clean/grounding/room-search3erfolgreichZiel getroffen6.79 s
smart-home-android/clean/grounding/thermostat-card3erfolgreichZiel getroffen7.32 s
smart-home-android/clean/grounding/add-device3erfolgreichZiel getroffen6.50 s
smart-home-android/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." }8.80 s
smart-home-android/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 355, "y": 508, "width": 645, "height": 70 }, "description": "The 'Kitchen lights' quick-view card is offset to the right and its right edge is clipped by the viewport, unlike the other aligned cards." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 355, "y": 508, "width": 645, "height": 70 }, "description": "The 'Kitchen lights' quick-view card is offset to the right and its right edge is clipped by the viewport, unlike the other aligned cards." }14.15 s
smart-home-android/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.64 s
smart-home-android/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }5.56 s
Muse Spark 1.3 · 144 Fallausführungen

Zurückgegebene Modellversionen: meta/muse-spark-1.3.

Ziele mit demselben Treffer-/Fehlerergebnis in allen drei Wiederholungen: 23 / 24. Übereinstimmung kann auch bedeuten, dass ein Ziel jedes Mal verfehlt wird.

FallWiederholungErgebnisAusgangZeit
commerce-catalog-desktop/clean/grounding/search1erfolgreichZiel getroffen5.13 s
commerce-catalog-desktop/clean/grounding/add-product1erfolgreichZiel getroffen2.39 s
commerce-catalog-desktop/clean/grounding/catalog-query1erfolgreichZiel getroffen3.01 s
commerce-catalog-desktop/clean/grounding/stock-filter1erfolgreichZiel getroffen6.57 s
commerce-catalog-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defects observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defects observed.", "has_issue": false, "region": null }4.09 s
commerce-catalog-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Product heading clipped on left edge showing 'IANAGEMENT' and 'ducts' instead of full title.", "has_issue": true, "region": { "height": 65, "width": 68, "x": 191, "y": 126 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Product heading clipped on left edge showing 'IANAGEMENT' and 'ducts' instead of full title.", "has_issue": true, "region": { "height": 65, "width": 68, "x": 191, "y": 126 } }7.80 s
commerce-catalog-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.66 s
commerce-catalog-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.79 s
analytics-dashboard-desktop/clean/grounding/audience-tab1erfolgreichZiel getroffen4.96 s
analytics-dashboard-desktop/clean/grounding/revenue-card1erfolgreichZiel getroffen5.08 s
analytics-dashboard-desktop/clean/grounding/search-field1erfolgreichZiel getroffen3.09 s
analytics-dashboard-desktop/clean/grounding/compare-toggle1erfolgreichZiel getroffen4.94 s
analytics-dashboard-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null }4.07 s
analytics-dashboard-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card title truncated to 'Monthly revenue across all sales chann', cutting off end of word.", "has_issue": true, "region": { "height": 30, "width": 330, "x": 200, "y": 248 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card title truncated to 'Monthly revenue across all sales chann', cutting off end of word.", "has_issue": true, "region": { "height": 30, "width": 330, "x": 200, "y": 248 } }8.22 s
analytics-dashboard-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.36 s
analytics-dashboard-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.71 s
iot-control-desktop/clean/grounding/devices-nav1erfolgreichZiel getroffen3.78 s
iot-control-desktop/clean/grounding/restart-device1erfolgreichZiel getroffen3.54 s
iot-control-desktop/clean/grounding/auto-mode1erfolgreichZiel getroffen6.48 s
iot-control-desktop/clean/grounding/pump-card1erfolgreichZiel getroffen5.57 s
iot-control-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null }8.45 s
iot-control-desktop/defect/layout-defect1erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null }5.07 s
iot-control-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.95 s
iot-control-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.71 s
banking-ios/clean/grounding/profile1erfolgreichZiel getroffen6.58 s
banking-ios/clean/grounding/transaction-search1erfolgreichZiel getroffen3.56 s
banking-ios/clean/grounding/hide-balance1erfolgreichZiel verfehlt8.77 s
banking-ios/clean/grounding/transfer1erfolgreichZiel getroffen3.70 s
banking-ios/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null }4.67 s
banking-ios/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "Header title 'banking' is cropped at left viewport edge with first letter partially cut off.", "has_issue": true, "region": { "height": 60, "width": 260, "x": 35, "y": 58 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "Header title 'banking' is cropped at left viewport edge with first letter partially cut off.", "has_issue": true, "region": { "height": 60, "width": 260, "x": 35, "y": 58 } }5.79 s
banking-ios/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.00 s
banking-ios/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.72 s
flight-pass-ios/clean/grounding/boarding-pass1erfolgreichZiel verfehlt4.66 s
flight-pass-ios/clean/grounding/check-in1erfolgreichZiel getroffen3.46 s
flight-pass-ios/clean/grounding/trips-tab1erfolgreichZiel getroffen5.15 s
flight-pass-ios/clean/grounding/profile-tab1erfolgreichZiel getroffen4.18 s
flight-pass-ios/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null }4.41 s
flight-pass-ios/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red Flight update banner overlaps blue banner beneath it, obscuring its text.", "has_issue": true, "region": { "height": 62, "width": 912, "x": 43, "y": 863 } } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red Flight update banner overlaps blue banner beneath it, obscuring its text.", "has_issue": true, "region": { "height": 62, "width": 912, "x": 43, "y": 863 } }9.53 s
flight-pass-ios/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.16 s
flight-pass-ios/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.53 s
smart-home-android/clean/grounding/more1erfolgreichZiel getroffen4.07 s
smart-home-android/clean/grounding/room-search1erfolgreichZiel getroffen5.98 s
smart-home-android/clean/grounding/thermostat-card1erfolgreichZiel getroffen4.82 s
smart-home-android/clean/grounding/add-device1erfolgreichZiel getroffen3.71 s
smart-home-android/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null }4.54 s
smart-home-android/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": "viewport_cropping", "description": "Kitchen lights Quick view card is offset right and clipped by viewport edge.", "has_issue": true, "region": { "height": 116, "width": 845, "x": 155, "y": 534 } } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": "viewport_cropping", "description": "Kitchen lights Quick view card is offset right and clipped by viewport edge.", "has_issue": true, "region": { "height": 116, "width": 845, "x": 155, "y": 534 } }8.58 s
smart-home-android/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.20 s
smart-home-android/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }3.74 s
commerce-catalog-desktop/clean/grounding/search2erfolgreichZiel getroffen4.15 s
commerce-catalog-desktop/clean/grounding/add-product2erfolgreichZiel getroffen3.72 s
commerce-catalog-desktop/clean/grounding/catalog-query2erfolgreichZiel getroffen2.27 s
commerce-catalog-desktop/clean/grounding/stock-filter2erfolgreichZiel getroffen5.52 s
commerce-catalog-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null }4.06 s
commerce-catalog-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Main heading clipped on left showing 'ducts' and 'IANAGEMENT' instead of full 'Products' title.", "has_issue": true, "region": { "height": 68, "width": 69, "x": 191, "y": 125 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Main heading clipped on left showing 'ducts' and 'IANAGEMENT' instead of full 'Products' title.", "has_issue": true, "region": { "height": 68, "width": 69, "x": 191, "y": 125 } }10.11 s
commerce-catalog-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.76 s
commerce-catalog-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }3.51 s
analytics-dashboard-desktop/clean/grounding/audience-tab2erfolgreichZiel getroffen4.69 s
analytics-dashboard-desktop/clean/grounding/revenue-card2erfolgreichZiel getroffen3.86 s
analytics-dashboard-desktop/clean/grounding/search-field2erfolgreichZiel getroffen4.04 s
analytics-dashboard-desktop/clean/grounding/compare-toggle2erfolgreichZiel getroffen7.95 s
analytics-dashboard-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null }4.03 s
analytics-dashboard-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card heading truncated at container edge, reading 'chann' instead of full text.", "has_issue": true, "region": { "height": 30, "width": 328, "x": 200, "y": 249 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card heading truncated at container edge, reading 'chann' instead of full text.", "has_issue": true, "region": { "height": 30, "width": 328, "x": 200, "y": 249 } }9.23 s
analytics-dashboard-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.09 s
analytics-dashboard-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.88 s
iot-control-desktop/clean/grounding/devices-nav2erfolgreichZiel getroffen4.10 s
iot-control-desktop/clean/grounding/restart-device2erfolgreichZiel getroffen3.19 s
iot-control-desktop/clean/grounding/auto-mode2erfolgreichZiel getroffen10.00 s
iot-control-desktop/clean/grounding/pump-card2erfolgreichZiel getroffen2.78 s
iot-control-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect detected.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect detected.", "has_issue": false, "region": null }5.68 s
iot-control-desktop/defect/layout-defect2erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No visible layout defects detected.", "has_issue": false, "region": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No visible layout defects detected.", "has_issue": false, "region": null }8.11 s
iot-control-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.31 s
iot-control-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.58 s
banking-ios/clean/grounding/profile2erfolgreichZiel getroffen4.57 s
banking-ios/clean/grounding/transaction-search2erfolgreichZiel getroffen4.25 s
banking-ios/clean/grounding/hide-balance2erfolgreichZiel getroffen8.61 s
banking-ios/clean/grounding/transfer2erfolgreichZiel getroffen3.44 s
banking-ios/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed", "has_issue": false, "region": null }3.40 s
banking-ios/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "The 'Banking' header text is cropped at the left viewport edge, cutting off the first letter.", "has_issue": true, "region": { "height": 55, "width": 250, "x": 30, "y": 80 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "The 'Banking' header text is cropped at the left viewport edge, cutting off the first letter.", "has_issue": true, "region": { "height": 55, "width": 250, "x": 30, "y": 80 } }5.50 s
banking-ios/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }8.35 s
banking-ios/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.99 s
flight-pass-ios/clean/grounding/boarding-pass2erfolgreichZiel verfehlt4.49 s
flight-pass-ios/clean/grounding/check-in2erfolgreichZiel getroffen4.15 s
flight-pass-ios/clean/grounding/trips-tab2erfolgreichZiel getroffen6.30 s
flight-pass-ios/clean/grounding/profile-tab2erfolgreichZiel getroffen5.09 s
flight-pass-ios/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null }4.56 s
flight-pass-ios/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red Flight update banner overlaps blue banner behind it, obscuring its text.", "has_issue": true, "region": { "height": 55, "width": 910, "x": 45, "y": 875 } } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red Flight update banner overlaps blue banner behind it, obscuring its text.", "has_issue": true, "region": { "height": 55, "width": 910, "x": 45, "y": 875 } }10.19 s
flight-pass-ios/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.61 s
flight-pass-ios/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.10 s
smart-home-android/clean/grounding/more2erfolgreichZiel getroffen6.58 s
smart-home-android/clean/grounding/room-search2erfolgreichZiel getroffen4.87 s
smart-home-android/clean/grounding/thermostat-card2erfolgreichZiel getroffen4.45 s
smart-home-android/clean/grounding/add-device2erfolgreichZiel getroffen4.54 s
smart-home-android/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect.", "has_issue": false, "region": null }4.85 s
smart-home-android/defect/layout-defect2erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": null, "description": "No layout defect detected", "has_issue": false, "region": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": null, "description": "No layout defect detected", "has_issue": false, "region": null }9.62 s
smart-home-android/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.87 s
smart-home-android/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.64 s
commerce-catalog-desktop/clean/grounding/search3erfolgreichZiel getroffen4.92 s
commerce-catalog-desktop/clean/grounding/add-product3erfolgreichZiel getroffen3.59 s
commerce-catalog-desktop/clean/grounding/catalog-query3erfolgreichZiel getroffen3.02 s
commerce-catalog-desktop/clean/grounding/stock-filter3erfolgreichZiel getroffen5.15 s
commerce-catalog-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect.", "has_issue": false, "region": null }4.52 s
commerce-catalog-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Page heading clipped on left edge showing 'ducts' instead of 'Products' with kicker text cut off.", "has_issue": true, "region": { "height": 68, "width": 69, "x": 190, "y": 124 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Page heading clipped on left edge showing 'ducts' instead of 'Products' with kicker text cut off.", "has_issue": true, "region": { "height": 68, "width": 69, "x": 190, "y": 124 } }5.27 s
commerce-catalog-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.90 s
commerce-catalog-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }4.55 s
analytics-dashboard-desktop/clean/grounding/audience-tab3erfolgreichZiel getroffen5.64 s
analytics-dashboard-desktop/clean/grounding/revenue-card3erfolgreichZiel getroffen3.06 s
analytics-dashboard-desktop/clean/grounding/search-field3erfolgreichZiel getroffen3.71 s
analytics-dashboard-desktop/clean/grounding/compare-toggle3erfolgreichZiel getroffen5.52 s
analytics-dashboard-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect detected.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect detected.", "has_issue": false, "region": null }3.62 s
analytics-dashboard-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card heading truncated to 'Monthly revenue across all sales chann', clipped at container edge.", "has_issue": true, "region": { "height": 30, "width": 330, "x": 200, "y": 250 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card heading truncated to 'Monthly revenue across all sales chann', clipped at container edge.", "has_issue": true, "region": { "height": 30, "width": 330, "x": 200, "y": 250 } }8.87 s
analytics-dashboard-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.07 s
analytics-dashboard-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.90 s
iot-control-desktop/clean/grounding/devices-nav3erfolgreichZiel getroffen3.49 s
iot-control-desktop/clean/grounding/restart-device3erfolgreichZiel getroffen4.07 s
iot-control-desktop/clean/grounding/auto-mode3erfolgreichZiel getroffen5.22 s
iot-control-desktop/clean/grounding/pump-card3erfolgreichZiel getroffen5.83 s
iot-control-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect detected.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect detected.", "has_issue": false, "region": null }4.75 s
iot-control-desktop/defect/layout-defect3erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No layout defect detected", "has_issue": false, "region": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No layout defect detected", "has_issue": false, "region": null }7.46 s
iot-control-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }4.12 s
iot-control-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.87 s
banking-ios/clean/grounding/profile3erfolgreichZiel getroffen4.00 s
banking-ios/clean/grounding/transaction-search3erfolgreichZiel getroffen5.64 s
banking-ios/clean/grounding/hide-balance3erfolgreichZiel getroffen6.21 s
banking-ios/clean/grounding/transfer3erfolgreichZiel getroffen3.53 s
banking-ios/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null }5.26 s
banking-ios/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "Header title Banking and top brand text truncated at left viewport edge with B cropped.", "has_issue": true, "region": { "height": 75, "width": 270, "x": 10, "y": 35 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "Header title Banking and top brand text truncated at left viewport edge with B cropped.", "has_issue": true, "region": { "height": 75, "width": 270, "x": 10, "y": 35 } }7.26 s
banking-ios/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.34 s
banking-ios/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }3.49 s
flight-pass-ios/clean/grounding/boarding-pass3erfolgreichZiel verfehlt3.76 s
flight-pass-ios/clean/grounding/check-in3erfolgreichZiel getroffen4.72 s
flight-pass-ios/clean/grounding/trips-tab3erfolgreichZiel getroffen4.97 s
flight-pass-ios/clean/grounding/profile-tab3erfolgreichZiel getroffen5.65 s
flight-pass-ios/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect detected.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect detected.", "has_issue": false, "region": null }5.01 s
flight-pass-ios/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red flight update banner overlaps blue banner behind it, obscuring its text", "has_issue": true, "region": { "height": 75, "width": 920, "x": 40, "y": 840 } } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red flight update banner overlaps blue banner behind it, obscuring its text", "has_issue": true, "region": { "height": 75, "width": 920, "x": 40, "y": 840 } }5.54 s
flight-pass-ios/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.05 s
flight-pass-ios/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.99 s
smart-home-android/clean/grounding/more3erfolgreichZiel getroffen10.73 s
smart-home-android/clean/grounding/room-search3erfolgreichZiel getroffen4.38 s
smart-home-android/clean/grounding/thermostat-card3erfolgreichZiel getroffen4.20 s
smart-home-android/clean/grounding/add-device3erfolgreichZiel getroffen3.82 s
smart-home-android/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defects observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defects observed.", "has_issue": false, "region": null }4.26 s
smart-home-android/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": "broken_responsive_reflow", "description": "The Kitchen lights card is indented and narrower than other cards, breaking list alignment.", "has_issue": true, "region": { "height": 90, "width": 675, "x": 280, "y": 530 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": "broken_responsive_reflow", "description": "The Kitchen lights card is indented and narrower than other cards, breaking list alignment.", "has_issue": true, "region": { "height": 90, "width": 675, "x": 280, "y": 530 } }11.76 s
smart-home-android/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.94 s
smart-home-android/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.33 s
GPT-5.6 Luna · 144 Fallausführungen

Zurückgegebene Modellversionen: openai/gpt-5.6-luna.

Ziele mit demselben Treffer-/Fehlerergebnis in allen drei Wiederholungen: 23 / 24. Übereinstimmung kann auch bedeuten, dass ein Ziel jedes Mal verfehlt wird.

FallWiederholungErgebnisAusgangZeit
commerce-catalog-desktop/clean/grounding/search1erfolgreichZiel getroffen2.56 s
commerce-catalog-desktop/clean/grounding/add-product1erfolgreichZiel getroffen2.00 s
commerce-catalog-desktop/clean/grounding/catalog-query1erfolgreichZiel getroffen2.45 s
commerce-catalog-desktop/clean/grounding/stock-filter1erfolgreichZiel getroffen2.89 s
commerce-catalog-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.03 s
commerce-catalog-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 127, "width": 70, "height": 62 }, "description": "The left portions of the page heading text are visibly clipped, showing “IAGEMENT” and “ducts” instead of the full labels." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 127, "width": 70, "height": 62 }, "description": "The left portions of the page heading text are visibly clipped, showing “IAGEMENT” and “ducts” instead of the full labels." }4.66 s
commerce-catalog-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.85 s
commerce-catalog-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.73 s
analytics-dashboard-desktop/clean/grounding/audience-tab1erfolgreichZiel getroffen5.28 s
analytics-dashboard-desktop/clean/grounding/revenue-card1erfolgreichZiel getroffen2.12 s
analytics-dashboard-desktop/clean/grounding/search-field1erfolgreichZiel getroffen1.87 s
analytics-dashboard-desktop/clean/grounding/compare-toggle1erfolgreichZiel getroffen3.44 s
analytics-dashboard-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.97 s
analytics-dashboard-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 250, "width": 326, "height": 34 }, "description": "The first summary card's heading is visibly truncated at the card's right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 250, "width": 326, "height": 34 }, "description": "The first summary card's heading is visibly truncated at the card's right edge." }3.08 s
analytics-dashboard-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.31 s
analytics-dashboard-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.29 s
iot-control-desktop/clean/grounding/devices-nav1erfolgreichZiel getroffen2.47 s
iot-control-desktop/clean/grounding/restart-device1erfolgreichZiel getroffen2.03 s
iot-control-desktop/clean/grounding/auto-mode1erfolgreichZiel getroffen4.02 s
iot-control-desktop/clean/grounding/pump-card1erfolgreichZiel getroffen2.64 s
iot-control-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.91 s
iot-control-desktop/defect/layout-defect1erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }3.79 s
iot-control-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.55 s
iot-control-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.29 s
banking-ios/clean/grounding/profile1erfolgreichZiel getroffen2.82 s
banking-ios/clean/grounding/transaction-search1erfolgreichZiel verfehlt1.87 s
banking-ios/clean/grounding/hide-balance1erfolgreichZiel getroffen3.78 s
banking-ios/clean/grounding/transfer1erfolgreichZiel getroffen1.69 s
banking-ios/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.95 s
banking-ios/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 44, "y": 71, "width": 244, "height": 33 }, "description": "The left edge of the large “Banking” heading is visibly cropped, leaving the initial character partially cut off." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 44, "y": 71, "width": 244, "height": 33 }, "description": "The left edge of the large “Banking” heading is visibly cropped, leaving the initial character partially cut off." }6.58 s
banking-ios/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.64 s
banking-ios/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.33 s
flight-pass-ios/clean/grounding/boarding-pass1erfolgreichZiel getroffen2.10 s
flight-pass-ios/clean/grounding/check-in1erfolgreichZiel getroffen2.62 s
flight-pass-ios/clean/grounding/trips-tab1erfolgreichZiel getroffen4.46 s
flight-pass-ios/clean/grounding/profile-tab1erfolgreichZiel getroffen5.20 s
flight-pass-ios/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.61 s
flight-pass-ios/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "A red flight update banner visibly covers the lower portion of a blue actionable button near the bottom of the screen." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "A red flight update banner visibly covers the lower portion of a blue actionable button near the bottom of the screen." }3.27 s
flight-pass-ios/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.40 s
flight-pass-ios/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.14 s
smart-home-android/clean/grounding/more1erfolgreichZiel getroffen2.51 s
smart-home-android/clean/grounding/room-search1erfolgreichZiel getroffen1.95 s
smart-home-android/clean/grounding/thermostat-card1erfolgreichZiel getroffen2.74 s
smart-home-android/clean/grounding/add-device1erfolgreichZiel getroffen2.44 s
smart-home-android/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.88 s
smart-home-android/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 510, "width": 600, "height": 69 }, "description": "The second Quick view card is visibly offset to the right and does not align with the other cards in the list." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 510, "width": 600, "height": 69 }, "description": "The second Quick view card is visibly offset to the right and does not align with the other cards in the list." }5.45 s
smart-home-android/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.57 s
smart-home-android/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.57 s
commerce-catalog-desktop/clean/grounding/search2erfolgreichZiel getroffen2.62 s
commerce-catalog-desktop/clean/grounding/add-product2erfolgreichZiel getroffen2.64 s
commerce-catalog-desktop/clean/grounding/catalog-query2erfolgreichZiel getroffen2.37 s
commerce-catalog-desktop/clean/grounding/stock-filter2erfolgreichZiel getroffen3.21 s
commerce-catalog-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.91 s
commerce-catalog-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 192, "y": 151, "width": 72, "height": 39 }, "description": "The page title is visibly clipped on the left, displaying only “ducts” instead of the full title." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 192, "y": 151, "width": 72, "height": 39 }, "description": "The page title is visibly clipped on the left, displaying only “ducts” instead of the full title." }4.33 s
commerce-catalog-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.17 s
commerce-catalog-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.70 s
analytics-dashboard-desktop/clean/grounding/audience-tab2erfolgreichZiel getroffen4.50 s
analytics-dashboard-desktop/clean/grounding/revenue-card2erfolgreichZiel getroffen2.67 s
analytics-dashboard-desktop/clean/grounding/search-field2erfolgreichZiel getroffen2.36 s
analytics-dashboard-desktop/clean/grounding/compare-toggle2erfolgreichZiel getroffen3.04 s
analytics-dashboard-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.84 s
analytics-dashboard-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 252, "width": 325, "height": 28 }, "description": "The first summary card's headline is visibly truncated at the right edge, ending mid-word." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 252, "width": 325, "height": 28 }, "description": "The first summary card's headline is visibly truncated at the right edge, ending mid-word." }2.88 s
analytics-dashboard-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.57 s
analytics-dashboard-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.66 s
iot-control-desktop/clean/grounding/devices-nav2erfolgreichZiel getroffen3.05 s
iot-control-desktop/clean/grounding/restart-device2erfolgreichZiel getroffen2.25 s
iot-control-desktop/clean/grounding/auto-mode2erfolgreichZiel getroffen2.80 s
iot-control-desktop/clean/grounding/pump-card2erfolgreichZiel getroffen3.40 s
iot-control-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.31 s
iot-control-desktop/defect/layout-defect2erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.85 s
iot-control-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.08 s
iot-control-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.07 s
banking-ios/clean/grounding/profile2erfolgreichZiel getroffen3.15 s
banking-ios/clean/grounding/transaction-search2erfolgreichZiel getroffen2.17 s
banking-ios/clean/grounding/hide-balance2erfolgreichZiel getroffen3.60 s
banking-ios/clean/grounding/transfer2erfolgreichZiel getroffen2.21 s
banking-ios/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.69 s
banking-ios/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 45, "y": 70, "width": 245, "height": 32 }, "description": "The left edge of the “Banking” header text is visibly clipped, with the initial letter cut off." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 45, "y": 70, "width": 245, "height": 32 }, "description": "The left edge of the “Banking” header text is visibly clipped, with the initial letter cut off." }4.84 s
banking-ios/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.59 s
banking-ios/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.55 s
flight-pass-ios/clean/grounding/boarding-pass2erfolgreichZiel getroffen3.22 s
flight-pass-ios/clean/grounding/check-in2erfolgreichZiel getroffen2.06 s
flight-pass-ios/clean/grounding/trips-tab2erfolgreichZiel getroffen7.16 s
flight-pass-ios/clean/grounding/profile-tab2erfolgreichZiel getroffen3.82 s
flight-pass-ios/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.55 s
flight-pass-ios/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 64 }, "description": "A red flight-update banner visibly overlaps and obscures the blue actionable button near the bottom of the screen." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 64 }, "description": "A red flight-update banner visibly overlaps and obscures the blue actionable button near the bottom of the screen." }3.37 s
flight-pass-ios/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.73 s
flight-pass-ios/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.64 s
smart-home-android/clean/grounding/more2erfolgreichZiel getroffen2.67 s
smart-home-android/clean/grounding/room-search2erfolgreichZiel getroffen2.30 s
smart-home-android/clean/grounding/thermostat-card2erfolgreichZiel getroffen2.26 s
smart-home-android/clean/grounding/add-device2erfolgreichZiel getroffen1.93 s
smart-home-android/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.57 s
smart-home-android/defect/layout-defect2erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.87 s
smart-home-android/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.17 s
smart-home-android/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.36 s
commerce-catalog-desktop/clean/grounding/search3erfolgreichZiel getroffen2.61 s
commerce-catalog-desktop/clean/grounding/add-product3erfolgreichZiel getroffen2.88 s
commerce-catalog-desktop/clean/grounding/catalog-query3erfolgreichZiel getroffen4.31 s
commerce-catalog-desktop/clean/grounding/stock-filter3erfolgreichZiel getroffen3.36 s
commerce-catalog-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.36 s
commerce-catalog-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 76, "height": 64 }, "description": "The page title is visibly clipped on the left, displaying “ducts” instead of the full “Products”." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 76, "height": 64 }, "description": "The page title is visibly clipped on the left, displaying “ducts” instead of the full “Products”." }3.66 s
commerce-catalog-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.69 s
commerce-catalog-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.80 s
analytics-dashboard-desktop/clean/grounding/audience-tab3erfolgreichZiel getroffen3.61 s
analytics-dashboard-desktop/clean/grounding/revenue-card3erfolgreichZiel getroffen2.33 s
analytics-dashboard-desktop/clean/grounding/search-field3erfolgreichZiel getroffen2.09 s
analytics-dashboard-desktop/clean/grounding/compare-toggle3erfolgreichZiel getroffen3.14 s
analytics-dashboard-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.33 s
analytics-dashboard-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 253, "width": 326, "height": 28 }, "description": "The first summary card's heading is visibly cut off at the right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 253, "width": 326, "height": 28 }, "description": "The first summary card's heading is visibly cut off at the right edge." }3.52 s
analytics-dashboard-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.61 s
analytics-dashboard-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.64 s
iot-control-desktop/clean/grounding/devices-nav3erfolgreichZiel getroffen2.44 s
iot-control-desktop/clean/grounding/restart-device3erfolgreichZiel getroffen3.00 s
iot-control-desktop/clean/grounding/auto-mode3erfolgreichZiel getroffen3.80 s
iot-control-desktop/clean/grounding/pump-card3erfolgreichZiel getroffen3.33 s
iot-control-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.57 s
iot-control-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 463, "y": 558, "width": 222, "height": 66 }, "description": "A maintenance notification overlays the device table content." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 463, "y": 558, "width": 222, "height": 66 }, "description": "A maintenance notification overlays the device table content." }3.65 s
iot-control-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.82 s
iot-control-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.48 s
banking-ios/clean/grounding/profile3erfolgreichZiel getroffen2.94 s
banking-ios/clean/grounding/transaction-search3erfolgreichZiel getroffen2.50 s
banking-ios/clean/grounding/hide-balance3erfolgreichZiel getroffen4.97 s
banking-ios/clean/grounding/transfer3erfolgreichZiel getroffen2.55 s
banking-ios/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.96 s
banking-ios/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 44, "y": 72, "width": 244, "height": 29 }, "description": "The left edge of the Banking heading is visibly clipped, cutting off the first character." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 44, "y": 72, "width": 244, "height": 29 }, "description": "The left edge of the Banking heading is visibly clipped, cutting off the first character." }3.82 s
banking-ios/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.26 s
banking-ios/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.30 s
flight-pass-ios/clean/grounding/boarding-pass3erfolgreichZiel getroffen2.52 s
flight-pass-ios/clean/grounding/check-in3erfolgreichZiel getroffen2.24 s
flight-pass-ios/clean/grounding/trips-tab3erfolgreichZiel getroffen4.28 s
flight-pass-ios/clean/grounding/profile-tab3erfolgreichZiel getroffen3.22 s
flight-pass-ios/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.70 s
flight-pass-ios/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 63 }, "description": "A red flight update banner obscures the lower portion of a blue actionable button near the bottom of the screen." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 63 }, "description": "A red flight update banner obscures the lower portion of a blue actionable button near the bottom of the screen." }3.14 s
flight-pass-ios/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.40 s
flight-pass-ios/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.32 s
smart-home-android/clean/grounding/more3erfolgreichZiel getroffen2.35 s
smart-home-android/clean/grounding/room-search3erfolgreichZiel getroffen2.65 s
smart-home-android/clean/grounding/thermostat-card3erfolgreichZiel getroffen2.64 s
smart-home-android/clean/grounding/add-device3erfolgreichZiel getroffen1.92 s
smart-home-android/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.88 s
smart-home-android/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 510, "width": 600, "height": 69 }, "description": "The second Quick view card is misaligned and compressed to the right compared with the other full-width cards." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 510, "width": 600, "height": 69 }, "description": "The second Quick view card is misaligned and compressed to the right compared with the other full-width cards." }4.82 s
smart-home-android/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.19 s
smart-home-android/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.62 s
Jev 1.14 · 144 Fallausführungen

Zurückgegebene Modellversionen: typesafe/jev-1.14-20261014.

Ziele mit demselben Treffer-/Fehlerergebnis in allen drei Wiederholungen: 24 / 24. Übereinstimmung kann auch bedeuten, dass ein Ziel jedes Mal verfehlt wird.

FallWiederholungErgebnisAusgangZeit
commerce-catalog-desktop/clean/grounding/search1erfolgreichZiel getroffen0.43 s
commerce-catalog-desktop/clean/grounding/add-product1erfolgreichZiel getroffen0.41 s
commerce-catalog-desktop/clean/grounding/catalog-query1erfolgreichZiel getroffen0.40 s
commerce-catalog-desktop/clean/grounding/stock-filter1erfolgreichZiel getroffen0.38 s
commerce-catalog-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.47 s
commerce-catalog-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 190, "y": 140, "width": 60, "height": 69.99999999999999 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 190, "y": 140, "width": 60, "height": 69.99999999999999 }, "description": "" }0.39 s
commerce-catalog-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.35 s
commerce-catalog-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.44 s
analytics-dashboard-desktop/clean/grounding/audience-tab1erfolgreichZiel getroffen0.53 s
analytics-dashboard-desktop/clean/grounding/revenue-card1erfolgreichZiel getroffen0.37 s
analytics-dashboard-desktop/clean/grounding/search-field1erfolgreichZiel getroffen0.38 s
analytics-dashboard-desktop/clean/grounding/compare-toggle1erfolgreichZiel getroffen0.34 s
analytics-dashboard-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.48 s
analytics-dashboard-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 360.00000000000006, "height": 110.00000000000001 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 360.00000000000006, "height": 110.00000000000001 }, "description": "" }0.36 s
analytics-dashboard-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.33 s
analytics-dashboard-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.39 s
iot-control-desktop/clean/grounding/devices-nav1erfolgreichZiel getroffen0.34 s
iot-control-desktop/clean/grounding/restart-device1erfolgreichZiel getroffen0.32 s
iot-control-desktop/clean/grounding/auto-mode1erfolgreichZiel getroffen0.41 s
iot-control-desktop/clean/grounding/pump-card1erfolgreichZiel getroffen0.37 s
iot-control-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.48 s
iot-control-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 450, "y": 550, "width": 239.99999999999994, "height": 89.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 450, "y": 550, "width": 239.99999999999994, "height": 89.99999999999997 }, "description": "" }0.38 s
iot-control-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.35 s
iot-control-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.46 s
banking-ios/clean/grounding/profile1erfolgreichZiel getroffen0.39 s
banking-ios/clean/grounding/transaction-search1erfolgreichZiel getroffen0.48 s
banking-ios/clean/grounding/hide-balance1erfolgreichZiel getroffen0.41 s
banking-ios/clean/grounding/transfer1erfolgreichZiel getroffen0.43 s
banking-ios/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.37 s
banking-ios/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" }0.30 s
banking-ios/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.37 s
banking-ios/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.38 s
flight-pass-ios/clean/grounding/boarding-pass1erfolgreichZiel getroffen0.31 s
flight-pass-ios/clean/grounding/check-in1erfolgreichZiel getroffen0.48 s
flight-pass-ios/clean/grounding/trips-tab1erfolgreichZiel getroffen0.34 s
flight-pass-ios/clean/grounding/profile-tab1erfolgreichZiel getroffen0.32 s
flight-pass-ios/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.30 s
flight-pass-ios/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 70.00000000000006 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 70.00000000000006 }, "description": "" }0.37 s
flight-pass-ios/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.28 s
flight-pass-ios/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.38 s
smart-home-android/clean/grounding/more1erfolgreichZiel getroffen0.36 s
smart-home-android/clean/grounding/room-search1erfolgreichZiel getroffen0.34 s
smart-home-android/clean/grounding/thermostat-card1erfolgreichZiel getroffen0.37 s
smart-home-android/clean/grounding/add-device1erfolgreichZiel getroffen0.33 s
smart-home-android/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.43 s
smart-home-android/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 470, "width": 590, "height": 99.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 470, "width": 590, "height": 99.99999999999997 }, "description": "" }0.39 s
smart-home-android/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.38 s
smart-home-android/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.41 s
commerce-catalog-desktop/clean/grounding/search2erfolgreichZiel getroffen0.44 s
commerce-catalog-desktop/clean/grounding/add-product2erfolgreichZiel getroffen0.42 s
commerce-catalog-desktop/clean/grounding/catalog-query2erfolgreichZiel getroffen0.38 s
commerce-catalog-desktop/clean/grounding/stock-filter2erfolgreichZiel getroffen0.42 s
commerce-catalog-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.38 s
commerce-catalog-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 180, "y": 140, "width": 70, "height": 69.99999999999999 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 180, "y": 140, "width": 70, "height": 69.99999999999999 }, "description": "" }0.39 s
commerce-catalog-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.38 s
commerce-catalog-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.54 s
analytics-dashboard-desktop/clean/grounding/audience-tab2erfolgreichZiel getroffen0.41 s
analytics-dashboard-desktop/clean/grounding/revenue-card2erfolgreichZiel getroffen0.66 s
analytics-dashboard-desktop/clean/grounding/search-field2erfolgreichZiel getroffen0.48 s
analytics-dashboard-desktop/clean/grounding/compare-toggle2erfolgreichZiel getroffen0.37 s
analytics-dashboard-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.39 s
analytics-dashboard-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 360.00000000000006, "height": 100 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 360.00000000000006, "height": 100 }, "description": "" }0.39 s
analytics-dashboard-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.36 s
analytics-dashboard-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.87 s
iot-control-desktop/clean/grounding/devices-nav2erfolgreichZiel getroffen0.46 s
iot-control-desktop/clean/grounding/restart-device2erfolgreichZiel getroffen0.66 s
iot-control-desktop/clean/grounding/auto-mode2erfolgreichZiel getroffen0.39 s
iot-control-desktop/clean/grounding/pump-card2erfolgreichZiel getroffen0.35 s
iot-control-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.33 s
iot-control-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 460, "y": 540, "width": 239.99999999999994, "height": 99.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 460, "y": 540, "width": 239.99999999999994, "height": 99.99999999999997 }, "description": "" }0.37 s
iot-control-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.33 s
iot-control-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.38 s
banking-ios/clean/grounding/profile2erfolgreichZiel getroffen0.33 s
banking-ios/clean/grounding/transaction-search2erfolgreichZiel getroffen0.30 s
banking-ios/clean/grounding/hide-balance2erfolgreichZiel getroffen0.33 s
banking-ios/clean/grounding/transfer2erfolgreichZiel getroffen0.41 s
banking-ios/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.30 s
banking-ios/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" }0.44 s
banking-ios/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.31 s
banking-ios/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.29 s
flight-pass-ios/clean/grounding/boarding-pass2erfolgreichZiel getroffen0.30 s
flight-pass-ios/clean/grounding/check-in2erfolgreichZiel getroffen0.29 s
flight-pass-ios/clean/grounding/trips-tab2erfolgreichZiel getroffen0.27 s
flight-pass-ios/clean/grounding/profile-tab2erfolgreichZiel getroffen0.31 s
flight-pass-ios/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.32 s
flight-pass-ios/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 70.00000000000006 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 70.00000000000006 }, "description": "" }0.31 s
flight-pass-ios/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.29 s
flight-pass-ios/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.29 s
smart-home-android/clean/grounding/more2erfolgreichZiel getroffen0.32 s
smart-home-android/clean/grounding/room-search2erfolgreichZiel getroffen0.30 s
smart-home-android/clean/grounding/thermostat-card2erfolgreichZiel getroffen0.38 s
smart-home-android/clean/grounding/add-device2erfolgreichZiel getroffen0.32 s
smart-home-android/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.29 s
smart-home-android/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 480, "width": 590, "height": 99.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 480, "width": 590, "height": 99.99999999999997 }, "description": "" }0.30 s
smart-home-android/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.32 s
smart-home-android/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.35 s
commerce-catalog-desktop/clean/grounding/search3erfolgreichZiel getroffen0.35 s
commerce-catalog-desktop/clean/grounding/add-product3erfolgreichZiel getroffen0.37 s
commerce-catalog-desktop/clean/grounding/catalog-query3erfolgreichZiel getroffen0.40 s
commerce-catalog-desktop/clean/grounding/stock-filter3erfolgreichZiel getroffen0.45 s
commerce-catalog-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.37 s
commerce-catalog-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 180, "y": 140, "width": 70, "height": 69.99999999999999 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 180, "y": 140, "width": 70, "height": 69.99999999999999 }, "description": "" }0.38 s
commerce-catalog-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.39 s
commerce-catalog-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.43 s
analytics-dashboard-desktop/clean/grounding/audience-tab3erfolgreichZiel getroffen0.33 s
analytics-dashboard-desktop/clean/grounding/revenue-card3erfolgreichZiel getroffen0.35 s
analytics-dashboard-desktop/clean/grounding/search-field3erfolgreichZiel getroffen0.36 s
analytics-dashboard-desktop/clean/grounding/compare-toggle3erfolgreichZiel getroffen0.38 s
analytics-dashboard-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.35 s
analytics-dashboard-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 350.00000000000006, "height": 110.00000000000001 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 350.00000000000006, "height": 110.00000000000001 }, "description": "" }0.36 s
analytics-dashboard-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.36 s
analytics-dashboard-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.39 s
iot-control-desktop/clean/grounding/devices-nav3erfolgreichZiel getroffen0.33 s
iot-control-desktop/clean/grounding/restart-device3erfolgreichZiel getroffen0.32 s
iot-control-desktop/clean/grounding/auto-mode3erfolgreichZiel getroffen0.34 s
iot-control-desktop/clean/grounding/pump-card3erfolgreichZiel getroffen0.35 s
iot-control-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.37 s
iot-control-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 460, "y": 550, "width": 229.99999999999991, "height": 89.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 460, "y": 550, "width": 229.99999999999991, "height": 89.99999999999997 }, "description": "" }0.37 s
iot-control-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.36 s
iot-control-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.44 s
banking-ios/clean/grounding/profile3erfolgreichZiel getroffen0.29 s
banking-ios/clean/grounding/transaction-search3erfolgreichZiel getroffen0.29 s
banking-ios/clean/grounding/hide-balance3erfolgreichZiel getroffen0.29 s
banking-ios/clean/grounding/transfer3erfolgreichZiel getroffen0.33 s
banking-ios/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.44 s
banking-ios/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" }0.30 s
banking-ios/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.28 s
banking-ios/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.44 s
flight-pass-ios/clean/grounding/boarding-pass3erfolgreichZiel getroffen0.32 s
flight-pass-ios/clean/grounding/check-in3erfolgreichZiel getroffen0.30 s
flight-pass-ios/clean/grounding/trips-tab3erfolgreichZiel getroffen0.29 s
flight-pass-ios/clean/grounding/profile-tab3erfolgreichZiel getroffen0.28 s
flight-pass-ios/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.36 s
flight-pass-ios/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 60.00000000000006 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 60.00000000000006 }, "description": "" }0.30 s
flight-pass-ios/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.29 s
flight-pass-ios/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.29 s
smart-home-android/clean/grounding/more3erfolgreichZiel getroffen0.29 s
smart-home-android/clean/grounding/room-search3erfolgreichZiel getroffen0.39 s
smart-home-android/clean/grounding/thermostat-card3erfolgreichZiel getroffen0.35 s
smart-home-android/clean/grounding/add-device3erfolgreichZiel getroffen0.35 s
smart-home-android/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.35 s
smart-home-android/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 480, "width": 590, "height": 99.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 480, "width": 590, "height": 99.99999999999997 }, "description": "" }0.53 s
smart-home-android/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.39 s
smart-home-android/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.34 s
GPT-6 Luna Decisions · 72 Fallausführungen

Zurückgegebene Modellversionen: openai/gpt-6-luna-decisions-20261006.

FallWiederholungErgebnisAusgangZeit
commerce-catalog-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.38 s
commerce-catalog-desktop/defect/layout-defect1erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null }0.45 s
commerce-catalog-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.32 s
commerce-catalog-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.35 s
analytics-dashboard-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.38 s
analytics-dashboard-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" }0.36 s
analytics-dashboard-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.30 s
analytics-dashboard-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.38 s
iot-control-desktop/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.32 s
iot-control-desktop/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" }0.52 s
iot-control-desktop/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.55 s
iot-control-desktop/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.26 s
banking-ios/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.24 s
banking-ios/defect/layout-defect1erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null }0.21 s
banking-ios/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.29 s
banking-ios/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.22 s
flight-pass-ios/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.53 s
flight-pass-ios/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" }0.66 s
flight-pass-ios/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.38 s
flight-pass-ios/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.19 s
smart-home-android/clean/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.33 s
smart-home-android/defect/layout-defect1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" }0.27 s
smart-home-android/clean/element-presence/present1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.26 s
smart-home-android/clean/element-presence/absent1erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.21 s
commerce-catalog-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.35 s
commerce-catalog-desktop/defect/layout-defect2erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null }0.38 s
commerce-catalog-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.27 s
commerce-catalog-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.36 s
analytics-dashboard-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.35 s
analytics-dashboard-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" }0.32 s
analytics-dashboard-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.47 s
analytics-dashboard-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.28 s
iot-control-desktop/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.74 s
iot-control-desktop/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" }0.60 s
iot-control-desktop/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.91 s
iot-control-desktop/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.41 s
banking-ios/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.24 s
banking-ios/defect/layout-defect2erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null }0.21 s
banking-ios/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.49 s
banking-ios/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.24 s
flight-pass-ios/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.62 s
flight-pass-ios/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" }0.27 s
flight-pass-ios/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.26 s
flight-pass-ios/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.23 s
smart-home-android/clean/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.24 s
smart-home-android/defect/layout-defect2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" }0.53 s
smart-home-android/clean/element-presence/present2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.29 s
smart-home-android/clean/element-presence/absent2erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.19 s
commerce-catalog-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.49 s
commerce-catalog-desktop/defect/layout-defect3erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null }0.25 s
commerce-catalog-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.50 s
commerce-catalog-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.48 s
analytics-dashboard-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.36 s
analytics-dashboard-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" }0.23 s
analytics-dashboard-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.32 s
analytics-dashboard-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.52 s
iot-control-desktop/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.36 s
iot-control-desktop/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" }0.37 s
iot-control-desktop/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.31 s
iot-control-desktop/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.28 s
banking-ios/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.25 s
banking-ios/defect/layout-defect3erfolgreichThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null }0.25 s
banking-ios/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.27 s
banking-ios/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.52 s
flight-pass-ios/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.54 s
flight-pass-ios/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" }1.22 s
flight-pass-ios/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.19 s
flight-pass-ios/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.89 s
smart-home-android/clean/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }1.09 s
smart-home-android/defect/layout-defect3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" }0.25 s
smart-home-android/clean/element-presence/present3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.21 s
smart-home-android/clean/element-presence/absent3erfolgreichThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.32 s

Der abgeschlossene Pilot

Erfasste Fallausführungen: 1368 / 1368. HTTP-Versuche: 1368. Gemeldete API-Kosten: $3.3802.

Neun Konfigurationen können Zielkoordinaten liefern. GPT-6 Luna Decisions beantwortet Klassifikationsfragen und ist von den Ergebnissen zur Ziellokalisierung ausgenommen.

Methodik

Messen, was das Modell tatsächlich sieht

Die Abschnitte zu den Experimenten dokumentieren ihre jeweiligen Bewertungsregeln. Diese gemeinsame Methodik behandelt den gemeinsamen Eingabevertrag, Provider-Abläufe, den Forschungskontext und die Grenzen.

EingabeNur Screenshot
Geometrie0–1000
DurchläufeDrei pro Fall
BewertungSeparate Metriken

Vorgänge

Antwortzeit, Fehler und Kosten

HTTP-Rundlauf und lokale Verarbeitung der abschließend gewerteten Versuche, ohne Abfragen der Generierungsmetadaten. Alle Übertragungsversuche fließen in die gemeldeten Kosten ein. Die reine Klassifikationskonfiguration führt halb so viele Fälle aus.

ModellAbgeschlossen / geplantEndgültige AnfragefehlerUngültige AntwortenMedianP95Gemeldete KostenKosten pro 1.000 Anfragen
GPT-6 Luna144 / 144002.18 s3.83 s$0.0187$0.1296
GPT-6.1 Sol144 / 144002.68 s4.88 s$0.2803$1.9468
Claude Sonnet 5.5144 / 144082.20 s3.99 s$0.5803$4.0297
Gemini 3.8 Flash144 / 144012.42 s10.96 s$0.2655$1.8437
Grok 4.7144 / 1440212.03 s28.38 s$1.1171$7.7580
Qwen3.8 Max (0902)144 / 144018.14 s14.76 s$0.6158$4.2762
Muse Spark 1.3144 / 144004.22 s9.18 s$0.4342$3.0151
GPT-5.6 Luna144 / 144002.34 s4.80 s$0.0440$0.3057
Jev 1.14144 / 144000.37 s0.48 s$0.0159$0.1104
GPT-6 Luna Decisions72 / 72000.34 s0.90 s$0.0085$0.1177

Sichtbarkeitsprüfungen: die gemeinsame Ja/Nein-Aufgabe

Vergleiche Antwortzeit und Kosten anhand derselben Sichtbarkeitsfälle. Layout-Anfragen an allgemeine Modelle verlangen zusätzlich einen Bereich. Der Arbeitsumfang unterscheidet sich daher von reinen Decisions-Klassifikationen.

ModellSichtbarkeitsfälleMedianP95Sichtbarkeitskosten pro 1.000 Anfragen
GPT-6 Luna361.45 s1.88 s$0.0752
GPT-6.1 Sol362.18 s2.68 s$1.4487
Claude Sonnet 5.5361.85 s2.78 s$2.9955
Gemini 3.8 Flash362.45 s2.98 s$1.2008
Grok 4.7361.97 s5.95 s$2.8797
Qwen3.8 Max (0902)364.51 s6.79 s$2.6708
Muse Spark 1.3362.06 s4.23 s$2.0156
GPT-5.6 Luna361.58 s2.11 s$0.1533
Jev 1.14360.37 s0.48 s$0.1069
GPT-6 Luna Decisions360.31 s0.89 s$0.1062

Methodik

Was sich geändert hat und was unverändert blieb

  • Die sechs Screenshots, Annotationen, Aufgaben-Prompts und ursprünglichen Antwortschemas stammen aus dem Korpus vom 13.08.2026. Die neue Edition enthält eigene Kopien.
  • Jeder Fall wird dreimal ausgeführt. Generative Modelle fordern eine niedrige Reasoning-Stufe und eine Ausgabe von bis zu 2.048 Tokens statt der ursprünglichen Einstellung von 400 Tokens an. Anbieter können darüber hinaus zusätzliche abrechenbare Reasoning-Tokens melden. Diese Änderung der Ausführungsbedingungen ist offengelegt.
  • Anfragen sind an verfügbare Standard-Endpunkte der jeweiligen Modellanbieter gebunden, ohne Ausweichrouting. Die genauen Endpunkte, Preise, Einstellungen und Hashes bleiben gespeichert.
  • Entscheidungsmodelle erhalten neben der Aufgabe native Fragen. Für Noul-Wahrscheinlichkeiten gilt ein fester Schwellenwert von 0,5. Jevs Klickpunkt wird aus der Boxmitte berechnet; andere Modelle mit Koordinatenausgabe sagen einen separaten Klickpunkt vorher.
  • Luna Decisions unterstützt Klassifikation, aber keine Box-Ausgabe. Seine Layout-Aufgabe misst daher nur das Vorhandensein und die Kategorie von Fehlern. Native typisierte Antworten und die Gültigkeit von Adapter-JSON sind nicht mit freier JSON-Generierung gleichzusetzen.
  • Ungültige Modellantworten werden weder korrigiert noch erneut ausgeführt. Nur vorübergehende Übertragungsfehler erlauben einen weiteren Versuch; alle Versuche bleiben erhalten. Schwankungen bei Anbietern und Netzwerk können die Antwortzeit beeinflussen. Gemeldete Kosten enthalten automatische Cache-Rabatte; spätere Wiederholungen können bereits gefüllte Caches nutzen.

Drei Wiederholungen messen die Konsistenz dieser Fälle. Sechs synthetische Screens belegen weder eine allgemeine Modellüberlegenheit noch Produktionszuverlässigkeit oder mehrstufige Navigation. Geräteaktionen, Fehlerbehebung und das Erreichen von Navigationszielen werden nicht geprüft.

Eingabevertrag

Nur Screenshot

Prompttext plus ein Base64-PNG – kein DOM, kein Barrierefreiheitsbaum, kein Quelltext, keine OCR, Tools, Plugins oder Websuche.

01

Festes Rendering

Chromium mit 1440 × 900 oder 390 × 844; Geräte-Skalierungsfaktor 1.

02

Normalisierte Geometrie

Browserbasierte Ground Truth in einem Koordinatenraum von 0–1000.

03

Stabile Paare

Fehlerfreie und fehlerhafte Varianten bewahren die Zielgeometrie und Capture-Hashes.

Vorgänge

Anbieter-Verhalten sichtbar halten

Fixierte Modell-IDs, direkte Anbieterrouten, deaktivierte Ausweichrouten und offengelegte aufgabenspezifische Antwortverträge.

Wiederholungsrichtlinie

Ein Wiederholungsversuch bei 429, 5xx oder Netzwerkfehlern – nicht bei falschen Antworten.

Betriebsmetriken

Latenz, Fehler, Wiederholungsversuche und Kosten bleiben von der Genauigkeit getrennt.

Datenschutz

Veröffentlichte Datensätze enthalten weder Schlüssel noch Anfragekennungen der Anbieter.

Daten und Lizenz

Diese Edition herunterladen

Modell-Vorprüfung ↓Ausführungsplan ↓Aufnahmemanifest ↓Ergebnisse und Einzelbewertungen ↓Erfasste Rohversuche ↓Erkenntnisse und Fehlerübersicht ↓

Von Repeato erstellte Aufnahmen und aggregierte Ergebnisse stehen unter CC BY 4.0. Für rohe Modellausgaben gelten weiterhin die Bedingungen der Anbieter. Beachte den Lizenzhinweis.

DE