Localiser précisément les commandes
GPT-6.1 Sol a atteint les 72 cibles sur trois répétitions, avec un chevauchement moyen de 81,2 % de la zone cible. Quatre configurations ont atteint chaque cible ; la précision des zones prédites variait.
Recherche Repeato · Benchmark LLM de vision
Peut-on faire confiance aux LLM multimodaux pour tester les interfaces utilisateur ?
Un benchmark d’automatisation des tests d’interface basé uniquement sur des captures d’écran : localiser les éléments interactifs, reconnaître l’absence d’un élément, détecter les défauts visibles de mise en page et mesurer le coût opérationnel d’une réponse.
2026-10-17
Six écrans déterministes. Vingt-quatre cibles à localiser. 10 configurations API. Trois répétitions par cas. Un pilote ciblé, pas un classement.
Écrans6
Cibles de repérage24
Catégories de défauts6
Configurations API10
Résultats du pilote
Ce que montre cette édition
Trois répétitions par cas sur six écrans synthétiques révèlent des différences à approfondir. Elles ne permettent ni classement général des modèles ni conclusion sur leur fiabilité en production.
Les vérifications de présence ont atteint le plafond
Les dix configurations ont répondu correctement aux 36 vérifications de présence. Ce petit ensemble vérifie le rejet des commandes absentes, mais ne distingue pas la précision des modèles.
Repérer un défaut n’est que la première étape
Jev a correctement identifié les 36 écrans intacts ou défectueux, mais a nommé le défaut exact dans 12 des 18 essais sur écrans défectueux. Ces essais répètent trois fois six défauts distincts.
Ce que cela signifie pour l’automatisation des tests
Utilisez le type de vision adapté à chaque étape du test
L’interaction précise, l’observation continue et l’interprétation sémantique répondent à des exigences de performance différentes. Il ne faut pas s’attendre à ce qu’une seule technique de vision les optimise toutes.
Observation localeJusqu’à 20 fois par secondeRequête à un modèle distantMédiane de 0,3 à 12 secondes dans ce piloteComment nous combinons la vision locale et les LLMRéduire l’analyse
La boucle locale de vision de Repeato peut inspecter l’écran à 20 Hz, soit toutes les 50 ms. Ce pilote a mesuré des temps de réponse distants médians d’environ 0,3 à 12 secondes selon la tâche. La tâche de présence commune permet une comparaison plus équitable ; une classification rapide ne signifie pas une localisation aussi rapide. Ces chiffres locaux et distants proviennent de mesures différentes, pas d’un test comparatif contrôlé.
Un retour rapide pour les interactions répétées
Les empreintes visuelles locales peuvent rechercher en continu les cibles enregistrées et agir dès que l’écran est prêt. Répéter des appels à distance au modèle à la même fréquence serait irréaliste.
Prévisible, privé et économique
La correspondance locale ne nécessite ni aller-retour réseau, ni disponibilité d’un fournisseur, ni frais d’API par requête. Les écrans restent sur le poste de travail, et le même algorithme peut fonctionner hors ligne.
L’IA pour les questions auxquelles la correspondance ne peut pas répondre
La vision par LLM justifie sa latence lorsqu’un test nécessite un jugement sémantique : interpréter du texte ou des nombres, comprendre du contenu dynamique ou vérifier des concepts comme une courbe ascendante.
Répartition pratique
Utilisez la vision par ordinateur locale pour la boucle de contrôle haute fréquence : trouver les cibles, attendre, interagir et comparer visuellement. Faites appel sélectivement à la vision par IA lorsque l’assertion nécessite réellement une compréhension linguistique ou sémantique.
Expériences
Trois tâches distinctes de vision d’interface utilisateur
L’ancrage, la présence des éléments et l’intégrité de l’interface utilisateur répondent à des questions différentes et utilisent des métriques distinctes. Le pilote ne les combine pas en un score global.
Expérience 1 · Ancrage de l’interface utilisateur
Un modèle peut-il localiser l’élément d’interface demandé ?
24 cibles × trois répétitions = 72 tentatives par configuration capable de renvoyer une géométrie. Les réponses invalides ou en échec comptent comme des cibles manquées.
Neuf configurations peuvent fournir des coordonnées cibles. GPT-6 Luna Decisions répond aux questions de classification et est exclu des résultats de localisation.
- GPT-6 Luna0.796
- GPT-6.1 Sol0.812
- Claude Sonnet 5.50.748
- Gemini 3.8 Flash0.733
- Grok 4.70.477
- Qwen3.8 Max (0902)0.760
- Muse Spark 1.30.630
- GPT-5.6 Luna0.786
- Jev 1.140.581
- GPT-6 Luna$0.0096
- GPT-6.1 Sol$0.1447
- Claude Sonnet 5.5$0.2970
- Gemini 3.8 Flash$0.1007
- Grok 4.7$0.6647
- Qwen3.8 Max (0902)$0.3254
- Muse Spark 1.3$0.2220
- GPT-5.6 Luna$0.0258
- Jev 1.14$0.0079
- GPT-6 Luna2.35 s
- GPT-6.1 Sol2.91 s
- Claude Sonnet 5.52.32 s
- Gemini 3.8 Flash2.75 s
- Grok 4.716.91 s
- Qwen3.8 Max (0902)8.89 s
- Muse Spark 1.34.80 s
- GPT-5.6 Luna2.93 s
- Jev 1.140.37 s
Compromis entre précision et coûtPlus haut et plus à gauche est préférable.
Méthodologie
Évaluation de l’ancrage de l’interface utilisateur
Quatre cibles par capture d’écran, couvrant les types courants d’éléments d’interface actionnables.
Exactitude des points
Le point de clic prédit se trouve à l’intérieur de la zone cible.
Exactitude des boîtes
IoU moyenne et part des prédictions avec une IoU ≥ 0,5.
Validité
Taux de réponses conformes au schéma ; trois répétitions par cas révèlent la cohérence de chaque cas.
Expérience 2 · Présence des éléments
Un modèle peut-il déterminer quand un élément est absent ?
Six cibles présentes et six absentes, chacune vérifiée trois fois. Ce test mesure la présence visible, pas l’accomplissement d’un objectif de navigation.
| 100 % valides36/36 cas · 0 essai échoué | 100% | 100% | 100% | 0% | 1.50 s | $0.0752pour 1 000 cas conservés | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100 % valides36/36 cas · 0 essai échoué | 100% | 100% | 100% | 0% | 2.24 s | $1.4487pour 1 000 cas conservés | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100 % valides36/36 cas · 0 essai échoué | 100% | 100% | 100% | 0% | 2.02 s | $2.9955pour 1 000 cas conservés | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100 % valides36/36 cas · 0 essai échoué | 100% | 100% | 100% | 0% | 2.44 s | $1.2008pour 1 000 cas conservés | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100 % valides36/36 cas · 0 essai échoué | 100% | 100% | 100% | 0% | 2.66 s | $2.8797pour 1 000 cas conservés | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100 % valides36/36 cas · 0 essai échoué | 100% | 100% | 100% | 0% | 4.81 s | $2.6708pour 1 000 cas conservés | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100 % valides36/36 cas · 0 essai échoué | 100% | 100% | 100% | 0% | 2.47 s | $2.0156pour 1 000 cas conservés | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100 % valides36/36 cas · 0 essai échoué | 100% | 100% | 100% | 0% | 1.59 s | $0.1533pour 1 000 cas conservés | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100 % valides36/36 cas · 0 essai échoué | 100% | 100% | 100% | 0% | 0.38 s | $0.1069pour 1 000 cas conservés | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100 % valides36/36 cas · 0 essai échoué | 100% | 100% | 100% | 0% | 0.39 s | $0.1062pour 1 000 cas conservés | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Trois répétitions par cas sur six écrans synthétiques. Les coûts incluent les remises de cache déclarées. Ces résultats n’établissent ni fiabilité en production ni classement général des modèles.
Méthodologie
Évaluation de la présence des éléments
Une requête dont la présence est avérée et une requête plausible mais absente par écran sain.
Détection
L’exactitude équilibrée pondère de manière égale les cas présents et absents.
Comportement de la classe
Le rappel, la spécificité et le taux de faux positifs révèlent la direction des erreurs.
Booléen uniquement
found=true ou found=false ; aucun score de localisation.
Expérience 3 · Intégrité de l’interface utilisateur
Un modèle peut-il distinguer les mises en page correctes des mises en page défectueuses ?
Six écrans sans défaut et six avec un défaut introduit × trois répétitions. Reconnaître un écran défectueux, nommer le défaut exact et localiser sa zone sont des résultats distincts.
Compromis entre précision de détection des défauts et coûtPlus haut et plus à gauche est préférable.
Compromis entre précision de détection des défauts et latencePlus haut et plus à gauche est préférable. La latence est le temps de réponse moyen par cas de défaut conservé. La variante Decisions limitée à la classification ne produit pas de régions.
| 100 % valides36/36 cas · 0 essai échoué | 92% | 83% | 100% | 56% | 3.04 s | $0.1751pour 1 000 cas conservés | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100 % valides36/36 cas · 0 essai échoué | 92% | 83% | 100% | 33% | 3.64 s | $2.3176pour 1 000 cas conservés | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Un passage par cas ; aucun intervalle de confiance ni affirmation de stabilité. Le coût total est extrapolé à 1 000 cas conservés afin que les exécutions incomplètes ne paraissent pas artificiellement moins coûteuses. Les lignes comportant moins de 12 réponses reposent sur un échantillon de cas plus réduit et ne sont pas directement comparables aux lignes complètes.36/36 cas · 0 essai échoué | 83% | 67% | 100% | 50% | 2.87 s | $4.8735pour 1 000 cas conservés | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 97% valides36/36 cas · 0 essai échoué | 78% | 100% | 56% | 72% | 7.06 s | $3.3765pour 1 000 cas conservés | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 97% valides36/36 cas · 0 essai échoué | 94% | 94% | 94% | 72% | 17.48 s | $9.6872pour 1 000 cas conservés | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 97% valides36/36 cas · 0 essai échoué | 94% | 94% | 94% | 39% | 11.78 s | $5.3957pour 1 000 cas conservés | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100 % valides36/36 cas · 0 essai échoué | 89% | 78% | 100% | 56% | 6.34 s | $3.8791pour 1 000 cas conservés | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100 % valides36/36 cas · 0 essai échoué | 92% | 83% | 100% | 67% | 2.88 s | $0.3518pour 1 000 cas conservés | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100 % valides36/36 cas · 0 essai échoué | 100% | 100% | 100% | 67% | 0.37 s | $0.1166pour 1 000 cas conservés | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 100 % valides36/36 cas · 0 essai échoué | 83% | 67% | 100% | 50% | 0.42 s | $0.1291pour 1 000 cas conservés | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Trois répétitions par cas sur six écrans synthétiques. Les coûts incluent les remises de cache déclarées. Ces résultats n’établissent ni fiabilité en production ni classement général des modèles.
Le calcul existant des défauts attribue un chevauchement nul lorsque la catégorie est incorrecte. Luna Decisions est évalué uniquement en classification et n’est pas tenu de produire une région.
Méthodologie
Évaluation de l’intégrité de l’interface utilisateur
Des scènes saines et défectueuses appariées couvrent six types objectifs de défaillances de mise en page.
Détection
Exactitude équilibrée, rappel, spécificité et taux de faux positifs.
Classification
Exactitude de la catégorie de défaut après détection réussie.
Localisation
IoU de la zone prédite par rapport aux régions de défaut dérivées du navigateur.
Corpus figé
Trois écrans de bureau et trois écrans mobiles
Chaque fixture comporte un contrôle propre et une variante délibérément défectueuse. Les captures utilisent des fenêtres d’affichage fixes, des ressources locales, une géométrie stable des cibles et des seuils exacts de différence entre pixels.
Fiches des modèles
Examiner chaque cas conservé
Chaque répétition reste visible, y compris les tentatives incorrectes, invalides et en échec. Dépliez un modèle pour examiner ses cas ; le téléchargement brut conserve les prédictions et les décisions natives.
openaiGPT-6 LunaRésultats complets du benchmark →openaiGPT-6.1 SolRésultats complets du benchmark →anthropicClaude Sonnet 5.5Résultats complets du benchmark →googleGemini 3.8 FlashRésultats complets du benchmark →x-aiGrok 4.7Résultats complets du benchmark →qwenQwen3.8 Max (0902)Résultats complets du benchmark →metaMuse Spark 1.3Résultats complets du benchmark →openaiGPT-5.6 LunaRésultats complets du benchmark →typesafeJev 1.14Résultats complets du benchmark →openaiGPT-6 Luna DecisionsRésultats complets du benchmark →
GPT-6 Luna · 144 exécutions de cas
Versions de modèle renvoyées : openai/gpt-6-luna.
Cibles avec le même résultat atteint/manqué sur les trois répétitions : 23 / 24. La concordance peut aussi signifier qu’une cible est manquée à chaque fois.
| Cas | Répétition | Résultat | Issue | Temps |
|---|---|---|---|---|
| commerce-catalog-desktop/clean/grounding/search | 1 | réussite | Cible atteinte | 3.33 s |
| commerce-catalog-desktop/clean/grounding/add-product | 1 | réussite | Cible atteinte | 1.92 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 1 | réussite | Cible atteinte | 2.65 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 1 | réussite | Cible atteinte | 3.77 s |
| commerce-catalog-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.33 s |
| commerce-catalog-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 191, "y": 145, "width": 69, "height": 46 }, "description": "The page heading is clipped on the left, showing only “ducts” instead of the full title." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 191, "y": 145, "width": 69, "height": 46 }, "description": "The page heading is clipped on the left, showing only “ducts” instead of the full title." } | 3.98 s |
| commerce-catalog-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.55 s |
| commerce-catalog-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.28 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 1 | réussite | Cible atteinte | 2.66 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 1 | réussite | Cible atteinte | 2.42 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 1 | réussite | Cible atteinte | 2.01 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 1 | réussite | Cible atteinte | 2.24 s |
| analytics-dashboard-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.99 s |
| analytics-dashboard-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 248, "width": 326, "height": 31 }, "description": "The revenue card heading is clipped at the card’s right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 248, "width": 326, "height": 31 }, "description": "The revenue card heading is clipped at the card’s right edge." } | 2.59 s |
| analytics-dashboard-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.57 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.57 s |
| iot-control-desktop/clean/grounding/devices-nav | 1 | réussite | Cible atteinte | 2.21 s |
| iot-control-desktop/clean/grounding/restart-device | 1 | réussite | Cible atteinte | 1.98 s |
| iot-control-desktop/clean/grounding/auto-mode | 1 | réussite | Cible atteinte | 2.62 s |
| iot-control-desktop/clean/grounding/pump-card | 1 | réussite | Cible atteinte | 2.80 s |
| iot-control-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.04 s |
| iot-control-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 222, "height": 64 }, "description": "The maintenance notification overlaps the device table rows." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 222, "height": 64 }, "description": "The maintenance notification overlaps the device table rows." } | 3.62 s |
| iot-control-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.49 s |
| iot-control-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.57 s |
| banking-ios/clean/grounding/profile | 1 | réussite | Cible atteinte | 2.46 s |
| banking-ios/clean/grounding/transaction-search | 1 | réussite | Cible atteinte | 2.81 s |
| banking-ios/clean/grounding/hide-balance | 1 | réussite | Cible atteinte | 2.59 s |
| banking-ios/clean/grounding/transfer | 1 | réussite | Cible atteinte | 1.87 s |
| banking-ios/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.16 s |
| banking-ios/defect/layout-defect | 1 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 6.07 s |
| banking-ios/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.72 s |
| banking-ios/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.24 s |
| flight-pass-ios/clean/grounding/boarding-pass | 1 | réussite | Cible atteinte | 2.49 s |
| flight-pass-ios/clean/grounding/check-in | 1 | réussite | Cible atteinte | 2.78 s |
| flight-pass-ios/clean/grounding/trips-tab | 1 | réussite | Cible atteinte | 1.63 s |
| flight-pass-ios/clean/grounding/profile-tab | 1 | réussite | Cible atteinte | 2.53 s |
| flight-pass-ios/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.76 s |
| flight-pass-ios/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "A red flight-update banner covers the lower portion of the blue action button." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "A red flight-update banner covers the lower portion of the blue action button." } | 2.79 s |
| flight-pass-ios/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.24 s |
| flight-pass-ios/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.12 s |
| smart-home-android/clean/grounding/more | 1 | réussite | Cible atteinte | 1.47 s |
| smart-home-android/clean/grounding/room-search | 1 | réussite | Cible atteinte | 1.88 s |
| smart-home-android/clean/grounding/thermostat-card | 1 | réussite | Cible atteinte | 1.46 s |
| smart-home-android/clean/grounding/add-device | 1 | réussite | Cible atteinte | 1.38 s |
| smart-home-android/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.93 s |
| smart-home-android/defect/layout-defect | 1 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 4.04 s |
| smart-home-android/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.19 s |
| smart-home-android/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.21 s |
| commerce-catalog-desktop/clean/grounding/search | 2 | réussite | Cible atteinte | 2.62 s |
| commerce-catalog-desktop/clean/grounding/add-product | 2 | réussite | Cible atteinte | 2.12 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 2 | réussite | Cible atteinte | 1.54 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 2 | réussite | Cible atteinte | 2.71 s |
| commerce-catalog-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.88 s |
| commerce-catalog-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 151, "width": 68, "height": 44 }, "description": "The page heading is clipped on the left; only “ducts” is visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 151, "width": 68, "height": 44 }, "description": "The page heading is clipped on the left; only “ducts” is visible." } | 3.44 s |
| commerce-catalog-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.75 s |
| commerce-catalog-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.65 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 2 | réussite | Cible atteinte | 3.18 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 2 | réussite | Cible atteinte | 2.89 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 2 | réussite | Cible atteinte | 2.21 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 2 | réussite | Cible atteinte | 2.67 s |
| analytics-dashboard-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.91 s |
| analytics-dashboard-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 253, "width": 326, "height": 37 }, "description": "The revenue card heading is cut off at the right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 253, "width": 326, "height": 37 }, "description": "The revenue card heading is cut off at the right edge." } | 2.65 s |
| analytics-dashboard-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.97 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.67 s |
| iot-control-desktop/clean/grounding/devices-nav | 2 | réussite | Cible atteinte | 2.20 s |
| iot-control-desktop/clean/grounding/restart-device | 2 | réussite | Cible atteinte | 2.25 s |
| iot-control-desktop/clean/grounding/auto-mode | 2 | réussite | Cible atteinte | 2.86 s |
| iot-control-desktop/clean/grounding/pump-card | 2 | réussite | Cible atteinte | 3.25 s |
| iot-control-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.92 s |
| iot-control-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 222, "height": 64 }, "description": "The maintenance notification overlaps the device table rows." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 222, "height": 64 }, "description": "The maintenance notification overlaps the device table rows." } | 3.14 s |
| iot-control-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.77 s |
| iot-control-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.64 s |
| banking-ios/clean/grounding/profile | 2 | réussite | Cible atteinte | 3.40 s |
| banking-ios/clean/grounding/transaction-search | 2 | réussite | Cible atteinte | 2.17 s |
| banking-ios/clean/grounding/hide-balance | 2 | réussite | Cible atteinte | 2.38 s |
| banking-ios/clean/grounding/transfer | 2 | réussite | Cible atteinte | 1.75 s |
| banking-ios/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.87 s |
| banking-ios/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 72, "width": 240, "height": 32 }, "description": "The first letter of the Banking heading is clipped at the left edge of its container." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 72, "width": 240, "height": 32 }, "description": "The first letter of the Banking heading is clipped at the left edge of its container." } | 5.97 s |
| banking-ios/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.17 s |
| banking-ios/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.45 s |
| flight-pass-ios/clean/grounding/boarding-pass | 2 | réussite | Cible atteinte | 2.26 s |
| flight-pass-ios/clean/grounding/check-in | 2 | réussite | Cible atteinte | 1.47 s |
| flight-pass-ios/clean/grounding/trips-tab | 2 | réussite | Cible atteinte | 1.62 s |
| flight-pass-ios/clean/grounding/profile-tab | 2 | réussite | Cible manquée | 1.55 s |
| flight-pass-ios/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.20 s |
| flight-pass-ios/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner obscures part of the blue actionable button beneath it." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner obscures part of the blue actionable button beneath it." } | 3.68 s |
| flight-pass-ios/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.22 s |
| flight-pass-ios/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.09 s |
| smart-home-android/clean/grounding/more | 2 | réussite | Cible atteinte | 1.72 s |
| smart-home-android/clean/grounding/room-search | 2 | réussite | Cible atteinte | 2.22 s |
| smart-home-android/clean/grounding/thermostat-card | 2 | réussite | Cible atteinte | 1.53 s |
| smart-home-android/clean/grounding/add-device | 2 | réussite | Cible atteinte | 1.48 s |
| smart-home-android/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.82 s |
| smart-home-android/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 354, "y": 510, "width": 646, "height": 69 }, "description": "The second Quick view card extends beyond the right edge of the viewport and is cropped." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 354, "y": 510, "width": 646, "height": 69 }, "description": "The second Quick view card extends beyond the right edge of the viewport and is cropped." } | 4.63 s |
| smart-home-android/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.18 s |
| smart-home-android/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.45 s |
| commerce-catalog-desktop/clean/grounding/search | 3 | réussite | Cible atteinte | 2.81 s |
| commerce-catalog-desktop/clean/grounding/add-product | 3 | réussite | Cible atteinte | 1.70 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 3 | réussite | Cible atteinte | 2.45 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 3 | réussite | Cible atteinte | 3.83 s |
| commerce-catalog-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 3.66 s |
| commerce-catalog-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 191, "y": 150, "width": 70, "height": 40 }, "description": "The page title is visibly clipped at its beginning, showing only “ducts”." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 191, "y": 150, "width": 70, "height": 40 }, "description": "The page title is visibly clipped at its beginning, showing only “ducts”." } | 3.67 s |
| commerce-catalog-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.85 s |
| commerce-catalog-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.33 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 3 | réussite | Cible atteinte | 3.45 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 3 | réussite | Cible atteinte | 2.06 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 3 | réussite | Cible atteinte | 2.00 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 3 | réussite | Cible atteinte | 2.30 s |
| analytics-dashboard-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.75 s |
| analytics-dashboard-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 253, "width": 326, "height": 32 }, "description": "The monthly revenue card heading is clipped at the card’s right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 253, "width": 326, "height": 32 }, "description": "The monthly revenue card heading is clipped at the card’s right edge." } | 3.12 s |
| analytics-dashboard-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.27 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.32 s |
| iot-control-desktop/clean/grounding/devices-nav | 3 | réussite | Cible atteinte | 2.50 s |
| iot-control-desktop/clean/grounding/restart-device | 3 | réussite | Cible atteinte | 2.75 s |
| iot-control-desktop/clean/grounding/auto-mode | 3 | réussite | Cible atteinte | 2.79 s |
| iot-control-desktop/clean/grounding/pump-card | 3 | réussite | Cible atteinte | 2.32 s |
| iot-control-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 3.03 s |
| iot-control-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 3.08 s |
| iot-control-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.44 s |
| iot-control-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.71 s |
| banking-ios/clean/grounding/profile | 3 | réussite | Cible atteinte | 2.41 s |
| banking-ios/clean/grounding/transaction-search | 3 | réussite | Cible atteinte | 2.90 s |
| banking-ios/clean/grounding/hide-balance | 3 | réussite | Cible atteinte | 2.29 s |
| banking-ios/clean/grounding/transfer | 3 | réussite | Cible atteinte | 2.51 s |
| banking-ios/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.21 s |
| banking-ios/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 0, "y": 69, "width": 285, "height": 35 }, "description": "The heading at the upper left is clipped on its left side." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 0, "y": 69, "width": 285, "height": 35 }, "description": "The heading at the upper left is clipped on its left side." } | 5.43 s |
| banking-ios/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.45 s |
| banking-ios/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.26 s |
| flight-pass-ios/clean/grounding/boarding-pass | 3 | réussite | Cible atteinte | 2.16 s |
| flight-pass-ios/clean/grounding/check-in | 3 | réussite | Cible atteinte | 2.23 s |
| flight-pass-ios/clean/grounding/trips-tab | 3 | réussite | Cible atteinte | 2.66 s |
| flight-pass-ios/clean/grounding/profile-tab | 3 | réussite | Cible atteinte | 3.25 s |
| flight-pass-ios/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.77 s |
| flight-pass-ios/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner obscures part of the blue actionable button beneath it." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner obscures part of the blue actionable button beneath it." } | 2.89 s |
| flight-pass-ios/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.24 s |
| flight-pass-ios/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.06 s |
| smart-home-android/clean/grounding/more | 3 | réussite | Cible atteinte | 2.34 s |
| smart-home-android/clean/grounding/room-search | 3 | réussite | Cible atteinte | 2.18 s |
| smart-home-android/clean/grounding/thermostat-card | 3 | réussite | Cible atteinte | 1.72 s |
| smart-home-android/clean/grounding/add-device | 3 | réussite | Cible atteinte | 1.51 s |
| smart-home-android/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.43 s |
| smart-home-android/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 354, "y": 510, "width": 646, "height": 69 }, "description": "The Kitchen lights card is cut off at the right edge of the viewport." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 354, "y": 510, "width": 646, "height": 69 }, "description": "The Kitchen lights card is cut off at the right edge of the viewport." } | 5.87 s |
| smart-home-android/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.47 s |
| smart-home-android/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.66 s |
GPT-6.1 Sol · 144 exécutions de cas
Versions de modèle renvoyées : openai/gpt-6.1-sol.
Cibles avec le même résultat atteint/manqué sur les trois répétitions : 24 / 24. La concordance peut aussi signifier qu’une cible est manquée à chaque fois.
| Cas | Répétition | Résultat | Issue | Temps |
|---|---|---|---|---|
| commerce-catalog-desktop/clean/grounding/search | 1 | réussite | Cible atteinte | 3.73 s |
| commerce-catalog-desktop/clean/grounding/add-product | 1 | réussite | Cible atteinte | 3.41 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 1 | réussite | Cible atteinte | 2.75 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 1 | réussite | Cible atteinte | 3.00 s |
| commerce-catalog-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 3.79 s |
| commerce-catalog-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 68, "height": 68 }, "description": "The page heading and its small label are clipped on the left, leaving only trailing portions of the text visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 68, "height": 68 }, "description": "The page heading and its small label are clipped on the left, leaving only trailing portions of the text visible." } | 5.65 s |
| commerce-catalog-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.04 s |
| commerce-catalog-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.52 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 1 | réussite | Cible atteinte | 4.02 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 1 | réussite | Cible atteinte | 3.04 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 1 | réussite | Cible atteinte | 3.42 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 1 | réussite | Cible atteinte | 3.78 s |
| analytics-dashboard-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.68 s |
| analytics-dashboard-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 250, "width": 326, "height": 31 }, "description": "The monthly revenue card heading is clipped at the card's right edge." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 250, "width": 326, "height": 31 }, "description": "The monthly revenue card heading is clipped at the card's right edge." } | 3.75 s |
| analytics-dashboard-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.18 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.13 s |
| iot-control-desktop/clean/grounding/devices-nav | 1 | réussite | Cible atteinte | 3.30 s |
| iot-control-desktop/clean/grounding/restart-device | 1 | réussite | Cible atteinte | 2.72 s |
| iot-control-desktop/clean/grounding/auto-mode | 1 | réussite | Cible atteinte | 2.67 s |
| iot-control-desktop/clean/grounding/pump-card | 1 | réussite | Cible atteinte | 2.61 s |
| iot-control-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.52 s |
| iot-control-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.67 s |
| iot-control-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.46 s |
| iot-control-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.27 s |
| banking-ios/clean/grounding/profile | 1 | réussite | Cible atteinte | 2.88 s |
| banking-ios/clean/grounding/transaction-search | 1 | réussite | Cible atteinte | 2.62 s |
| banking-ios/clean/grounding/hide-balance | 1 | réussite | Cible atteinte | 2.78 s |
| banking-ios/clean/grounding/transfer | 1 | réussite | Cible atteinte | 3.09 s |
| banking-ios/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.96 s |
| banking-ios/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 47, "width": 238, "height": 59 }, "description": "The brand label and Banking heading are clipped along their left edge, cutting off their initial letters." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 47, "width": 238, "height": 59 }, "description": "The brand label and Banking heading are clipped along their left edge, cutting off their initial letters." } | 8.64 s |
| banking-ios/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.87 s |
| banking-ios/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.35 s |
| flight-pass-ios/clean/grounding/boarding-pass | 1 | réussite | Cible atteinte | 2.48 s |
| flight-pass-ios/clean/grounding/check-in | 1 | réussite | Cible atteinte | 4.77 s |
| flight-pass-ios/clean/grounding/trips-tab | 1 | réussite | Cible atteinte | 2.43 s |
| flight-pass-ios/clean/grounding/profile-tab | 1 | réussite | Cible atteinte | 2.85 s |
| flight-pass-ios/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.39 s |
| flight-pass-ios/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 64 }, "description": "The flight update banner overlaps the bottom blue action button, obscuring its label and lower portion." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 64 }, "description": "The flight update banner overlaps the bottom blue action button, obscuring its label and lower portion." } | 3.86 s |
| flight-pass-ios/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.98 s |
| flight-pass-ios/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.53 s |
| smart-home-android/clean/grounding/more | 1 | réussite | Cible atteinte | 2.27 s |
| smart-home-android/clean/grounding/room-search | 1 | réussite | Cible atteinte | 2.44 s |
| smart-home-android/clean/grounding/thermostat-card | 1 | réussite | Cible atteinte | 2.78 s |
| smart-home-android/clean/grounding/add-device | 1 | réussite | Cible atteinte | 2.72 s |
| smart-home-android/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.44 s |
| smart-home-android/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 70 }, "description": "The Kitchen lights card is shifted right and clipped at the list's right edge, cutting off its right side and trailing control." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 70 }, "description": "The Kitchen lights card is shifted right and clipped at the list's right edge, cutting off its right side and trailing control." } | 5.47 s |
| smart-home-android/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.67 s |
| smart-home-android/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.36 s |
| commerce-catalog-desktop/clean/grounding/search | 2 | réussite | Cible atteinte | 2.80 s |
| commerce-catalog-desktop/clean/grounding/add-product | 2 | réussite | Cible atteinte | 2.61 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 2 | réussite | Cible atteinte | 2.82 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 2 | réussite | Cible atteinte | 3.15 s |
| commerce-catalog-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 3.00 s |
| commerce-catalog-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 73, "height": 68 }, "description": "The page heading and its section label are clipped on the left, hiding their initial characters." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 73, "height": 68 }, "description": "The page heading and its section label are clipped on the left, hiding their initial characters." } | 4.90 s |
| commerce-catalog-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.41 s |
| commerce-catalog-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.60 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 2 | réussite | Cible atteinte | 2.80 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 2 | réussite | Cible atteinte | 2.48 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 2 | réussite | Cible atteinte | 2.85 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 2 | réussite | Cible atteinte | 3.92 s |
| analytics-dashboard-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.98 s |
| analytics-dashboard-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 249, "width": 326, "height": 32 }, "description": "The monthly revenue heading is cut off at the right edge of its card." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 249, "width": 326, "height": 32 }, "description": "The monthly revenue heading is cut off at the right edge of its card." } | 3.47 s |
| analytics-dashboard-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.60 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.42 s |
| iot-control-desktop/clean/grounding/devices-nav | 2 | réussite | Cible atteinte | 3.34 s |
| iot-control-desktop/clean/grounding/restart-device | 2 | réussite | Cible atteinte | 2.94 s |
| iot-control-desktop/clean/grounding/auto-mode | 2 | réussite | Cible atteinte | 4.14 s |
| iot-control-desktop/clean/grounding/pump-card | 2 | réussite | Cible atteinte | 2.51 s |
| iot-control-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.47 s |
| iot-control-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.90 s |
| iot-control-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.94 s |
| iot-control-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.50 s |
| banking-ios/clean/grounding/profile | 2 | réussite | Cible atteinte | 2.89 s |
| banking-ios/clean/grounding/transaction-search | 2 | réussite | Cible atteinte | 2.58 s |
| banking-ios/clean/grounding/hide-balance | 2 | réussite | Cible atteinte | 2.40 s |
| banking-ios/clean/grounding/transfer | 2 | réussite | Cible atteinte | 2.81 s |
| banking-ios/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.89 s |
| banking-ios/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 69, "width": 241, "height": 38 }, "description": "The left edge of the Banking heading is clipped, cutting off part of its first letter." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 69, "width": 241, "height": 38 }, "description": "The left edge of the Banking heading is clipped, cutting off part of its first letter." } | 7.42 s |
| banking-ios/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.72 s |
| banking-ios/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.02 s |
| flight-pass-ios/clean/grounding/boarding-pass | 2 | réussite | Cible atteinte | 3.14 s |
| flight-pass-ios/clean/grounding/check-in | 2 | réussite | Cible atteinte | 2.86 s |
| flight-pass-ios/clean/grounding/trips-tab | 2 | réussite | Cible atteinte | 2.24 s |
| flight-pass-ios/clean/grounding/profile-tab | 2 | réussite | Cible atteinte | 2.47 s |
| flight-pass-ios/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.99 s |
| flight-pass-ios/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 64 }, "description": "The red flight-update banner overlaps the blue bottom action button, obscuring its label and much of its clickable area." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 64 }, "description": "The red flight-update banner overlaps the blue bottom action button, obscuring its label and much of its clickable area." } | 4.43 s |
| flight-pass-ios/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.94 s |
| flight-pass-ios/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.89 s |
| smart-home-android/clean/grounding/more | 2 | réussite | Cible atteinte | 2.35 s |
| smart-home-android/clean/grounding/room-search | 2 | réussite | Cible atteinte | 2.58 s |
| smart-home-android/clean/grounding/thermostat-card | 2 | réussite | Cible atteinte | 2.82 s |
| smart-home-android/clean/grounding/add-device | 2 | réussite | Cible atteinte | 2.71 s |
| smart-home-android/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.13 s |
| smart-home-android/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 71 }, "description": "The Kitchen lights row is shifted right and clipped at the content boundary, cutting off its right edge and trailing control." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 71 }, "description": "The Kitchen lights row is shifted right and clipped at the content boundary, cutting off its right edge and trailing control." } | 5.02 s |
| smart-home-android/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.05 s |
| smart-home-android/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.98 s |
| commerce-catalog-desktop/clean/grounding/search | 3 | réussite | Cible atteinte | 3.93 s |
| commerce-catalog-desktop/clean/grounding/add-product | 3 | réussite | Cible atteinte | 2.40 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 3 | réussite | Cible atteinte | 3.39 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 3 | réussite | Cible atteinte | 2.85 s |
| commerce-catalog-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the specified categories." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the specified categories." } | 3.40 s |
| commerce-catalog-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 70, "height": 68 }, "description": "The page heading and its overline are clipped on the left, leaving only trailing portions of the text visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 70, "height": 68 }, "description": "The page heading and its overline are clipped on the left, leaving only trailing portions of the text visible." } | 4.58 s |
| commerce-catalog-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.23 s |
| commerce-catalog-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 3.35 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 3 | réussite | Cible atteinte | 2.94 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 3 | réussite | Cible atteinte | 2.86 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 3 | réussite | Cible atteinte | 2.81 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 3 | réussite | Cible atteinte | 3.48 s |
| analytics-dashboard-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.59 s |
| analytics-dashboard-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 249, "width": 327, "height": 32 }, "description": "The monthly revenue heading is cut off at the right edge of its card." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 249, "width": 327, "height": 32 }, "description": "The monthly revenue heading is cut off at the right edge of its card." } | 3.59 s |
| analytics-dashboard-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.55 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.03 s |
| iot-control-desktop/clean/grounding/devices-nav | 3 | réussite | Cible atteinte | 3.04 s |
| iot-control-desktop/clean/grounding/restart-device | 3 | réussite | Cible atteinte | 3.26 s |
| iot-control-desktop/clean/grounding/auto-mode | 3 | réussite | Cible atteinte | 2.99 s |
| iot-control-desktop/clean/grounding/pump-card | 3 | réussite | Cible atteinte | 2.80 s |
| iot-control-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 3.12 s |
| iot-control-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.84 s |
| iot-control-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.39 s |
| iot-control-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.19 s |
| banking-ios/clean/grounding/profile | 3 | réussite | Cible atteinte | 3.35 s |
| banking-ios/clean/grounding/transaction-search | 3 | réussite | Cible atteinte | 2.37 s |
| banking-ios/clean/grounding/hide-balance | 3 | réussite | Cible atteinte | 3.37 s |
| banking-ios/clean/grounding/transfer | 3 | réussite | Cible atteinte | 2.54 s |
| banking-ios/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.05 s |
| banking-ios/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 44, "y": 68, "width": 245, "height": 39 }, "description": "The left edge of the Banking heading is clipped, cutting off part of its first letter." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 44, "y": 68, "width": 245, "height": 39 }, "description": "The left edge of the Banking heading is clipped, cutting off part of its first letter." } | 6.93 s |
| banking-ios/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.85 s |
| banking-ios/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.60 s |
| flight-pass-ios/clean/grounding/boarding-pass | 3 | réussite | Cible atteinte | 2.12 s |
| flight-pass-ios/clean/grounding/check-in | 3 | réussite | Cible atteinte | 2.21 s |
| flight-pass-ios/clean/grounding/trips-tab | 3 | réussite | Cible atteinte | 3.53 s |
| flight-pass-ios/clean/grounding/profile-tab | 3 | réussite | Cible atteinte | 2.35 s |
| flight-pass-ios/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.03 s |
| flight-pass-ios/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner covers the lower portion and label of the blue action button above the bottom navigation." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner covers the lower portion and label of the blue action button above the bottom navigation." } | 4.19 s |
| flight-pass-ios/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.91 s |
| flight-pass-ios/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.12 s |
| smart-home-android/clean/grounding/more | 3 | réussite | Cible atteinte | 2.30 s |
| smart-home-android/clean/grounding/room-search | 3 | réussite | Cible atteinte | 2.60 s |
| smart-home-android/clean/grounding/thermostat-card | 3 | réussite | Cible atteinte | 2.75 s |
| smart-home-android/clean/grounding/add-device | 3 | réussite | Cible atteinte | 2.49 s |
| smart-home-android/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.17 s |
| smart-home-android/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 70 }, "description": "The Kitchen lights card is shifted right and clipped at the list container’s right edge, cutting off its right side and trailing control." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 70 }, "description": "The Kitchen lights card is shifted right and clipped at the list container’s right edge, cutting off its right side and trailing control." } | 5.17 s |
| smart-home-android/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.06 s |
| smart-home-android/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.07 s |
Claude Sonnet 5.5 · 144 exécutions de cas
Versions de modèle renvoyées : anthropic/claude-sonnet-5.5.
Cibles avec le même résultat atteint/manqué sur les trois répétitions : 22 / 24. La concordance peut aussi signifier qu’une cible est manquée à chaque fois.
| Cas | Répétition | Résultat | Issue | Temps |
|---|---|---|---|---|
| commerce-catalog-desktop/clean/grounding/search | 1 | réussite | Cible atteinte | 3.96 s |
| commerce-catalog-desktop/clean/grounding/add-product | 1 | réponse invalide | Cible manquée | 2.02 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 1 | réussite | Cible atteinte | 3.08 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 1 | réussite | Cible atteinte | 2.40 s |
| commerce-catalog-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact and aligned." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact and aligned." } | 3.75 s |
| commerce-catalog-desktop/defect/layout-defect | 1 | réponse invalide | The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 191, "y": 123, "width": 200, "height": 75 }, "description": "The page heading and eyebrow text are cut off on the left edge. The eyebrow reads 'IANAGEMENT' instead of 'MANAGEMENT', and the title reads 'ducts' instead of 'Products', so the leading characters are clipped by the container." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 191, "y": 123, "width": 200, "height": 75 }, "description": "The page heading and eyebrow text are cut off on the left edge. The eyebrow reads 'IANAGEMENT' instead of 'MANAGEMENT', and the title reads 'ducts' instead of 'Products', so the leading characters are clipped by the container." } | 4.65 s |
| commerce-catalog-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.46 s |
| commerce-catalog-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.24 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 1 | réussite | Cible atteinte | 2.47 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 1 | réussite | Cible atteinte | 2.46 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 1 | réussite | Cible atteinte | 2.94 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 1 | réussite | Cible atteinte | 2.31 s |
| analytics-dashboard-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible clipping, overflow, overlap, obscured controls, or reflow problems; all text and controls are fully visible and properly contained." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible clipping, overflow, overlap, obscured controls, or reflow problems; all text and controls are fully visible and properly contained." } | 4.48 s |
| analytics-dashboard-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 250, "width": 356, "height": 32 }, "description": "The red card title 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is clipped by the container." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 250, "width": 356, "height": 32 }, "description": "The red card title 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is clipped by the container." } | 2.55 s |
| analytics-dashboard-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.14 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.14 s |
| iot-control-desktop/clean/grounding/devices-nav | 1 | réussite | Cible atteinte | 3.05 s |
| iot-control-desktop/clean/grounding/restart-device | 1 | réussite | Cible atteinte | 2.13 s |
| iot-control-desktop/clean/grounding/auto-mode | 1 | réussite | Cible atteinte | 2.39 s |
| iot-control-desktop/clean/grounding/pump-card | 1 | réussite | Cible atteinte | 2.31 s |
| iot-control-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact." } | 3.98 s |
| iot-control-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' banner floats over the table rows, covering part of the Cooling pump and Intake valve row areas and overlapping the row divider." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' banner floats over the table rows, covering part of the Cooling pump and Intake valve row areas and overlapping the row divider." } | 4.73 s |
| iot-control-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.99 s |
| iot-control-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.35 s |
| banking-ios/clean/grounding/profile | 1 | réussite | Cible atteinte | 1.84 s |
| banking-ios/clean/grounding/transaction-search | 1 | réussite | Cible atteinte | 3.14 s |
| banking-ios/clean/grounding/hide-balance | 1 | réussite | Cible atteinte | 1.92 s |
| banking-ios/clean/grounding/transfer | 1 | réussite | Cible atteinte | 1.82 s |
| banking-ios/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all text and controls are fully visible, contained, and not overlapping." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all text and controls are fully visible, contained, and not overlapping." } | 2.21 s |
| banking-ios/defect/layout-defect | 1 | réponse invalide | The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 300, "height": 115 }, "description": "The header title is cut off at the left screen edge: 'banking' renders as 'oanking' and the 'MAYA' label above it as 'IAYA', so the leading characters are cropped." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 300, "height": 115 }, "description": "The header title is cut off at the left screen edge: 'banking' renders as 'oanking' and the 'MAYA' label above it as 'IAYA', so the leading characters are cropped." } | 3.40 s |
| banking-ios/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.51 s |
| banking-ios/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.83 s |
| flight-pass-ios/clean/grounding/boarding-pass | 1 | réussite | Cible atteinte | 2.46 s |
| flight-pass-ios/clean/grounding/check-in | 1 | réussite | Cible atteinte | 1.91 s |
| flight-pass-ios/clean/grounding/trips-tab | 1 | réussite | Cible atteinte | 1.89 s |
| flight-pass-ios/clean/grounding/profile-tab | 1 | réussite | Cible atteinte | 2.30 s |
| flight-pass-ios/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; content is fully visible, text fits its containers, and no elements overlap or obscure controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; content is fully visible, text fits its containers, and no elements overlap or obscure controls." } | 2.34 s |
| flight-pass-ios/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 80 }, "description": "A red 'Flight update' banner covers the blue primary button near the bottom, hiding most of its label so only the tops of the letters remain visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 80 }, "description": "A red 'Flight update' banner covers the blue primary button near the bottom, hiding most of its label so only the tops of the letters remain visible." } | 3.21 s |
| flight-pass-ios/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.77 s |
| flight-pass-ios/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 4.33 s |
| smart-home-android/clean/grounding/more | 1 | réussite | Cible atteinte | 2.23 s |
| smart-home-android/clean/grounding/room-search | 1 | réussite | Cible atteinte | 9.81 s |
| smart-home-android/clean/grounding/thermostat-card | 1 | réussite | Cible atteinte | 2.25 s |
| smart-home-android/clean/grounding/add-device | 1 | réussite | Cible atteinte | 1.83 s |
| smart-home-android/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; all elements are fully rendered and aligned." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; all elements are fully rendered and aligned." } | 4.38 s |
| smart-home-android/defect/layout-defect | 1 | réponse invalide | The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 533, "width": 646, "height": 46 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its right side is cut off." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 533, "width": 646, "height": 46 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its right side is cut off." } | 5.42 s |
| smart-home-android/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.53 s |
| smart-home-android/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.99 s |
| commerce-catalog-desktop/clean/grounding/search | 2 | réussite | Cible atteinte | 2.13 s |
| commerce-catalog-desktop/clean/grounding/add-product | 2 | réussite | Cible atteinte | 1.89 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 2 | réussite | Cible atteinte | 2.72 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 2 | réussite | Cible atteinte | 2.05 s |
| commerce-catalog-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." } | 2.41 s |
| commerce-catalog-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 125, "width": 205, "height": 57 }, "description": "The page heading and eyebrow text are cut off at the left edge of their container, showing 'IANAGEMENT' and 'ducts' instead of the full words." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 125, "width": 205, "height": 57 }, "description": "The page heading and eyebrow text are cut off at the left edge of their container, showing 'IANAGEMENT' and 'ducts' instead of the full words." } | 3.99 s |
| commerce-catalog-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.64 s |
| commerce-catalog-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.47 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 2 | réussite | Cible atteinte | 2.29 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 2 | réussite | Cible atteinte | 2.15 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 2 | réussite | Cible atteinte | 2.47 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 2 | réussite | Cible atteinte | 2.02 s |
| analytics-dashboard-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible. The layout appears intact." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible. The layout appears intact." } | 3.55 s |
| analytics-dashboard-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 249, "width": 355, "height": 40 }, "description": "The red heading 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is clipped by the container." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 249, "width": 355, "height": 40 }, "description": "The red heading 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is clipped by the container." } | 2.22 s |
| analytics-dashboard-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.77 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.68 s |
| iot-control-desktop/clean/grounding/devices-nav | 2 | réussite | Cible atteinte | 2.64 s |
| iot-control-desktop/clean/grounding/restart-device | 2 | réussite | Cible atteinte | 2.20 s |
| iot-control-desktop/clean/grounding/auto-mode | 2 | réussite | Cible atteinte | 2.46 s |
| iot-control-desktop/clean/grounding/pump-card | 2 | réussite | Cible atteinte | 2.21 s |
| iot-control-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." } | 2.09 s |
| iot-control-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "A red 'Maintenance scheduled' banner floats over the device table, covering parts of the Cooling pump and Intake valve rows and overlapping the row divider." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "A red 'Maintenance scheduled' banner floats over the device table, covering parts of the Cooling pump and Intake valve rows and overlapping the row divider." } | 3.48 s |
| iot-control-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.35 s |
| iot-control-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.89 s |
| banking-ios/clean/grounding/profile | 2 | réussite | Cible atteinte | 2.23 s |
| banking-ios/clean/grounding/transaction-search | 2 | réussite | Cible atteinte | 1.61 s |
| banking-ios/clean/grounding/hide-balance | 2 | réussite | Cible atteinte | 1.87 s |
| banking-ios/clean/grounding/transfer | 2 | réussite | Cible atteinte | 1.83 s |
| banking-ios/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all text, cards, and controls render fully within their containers without overlap or cropping." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all text, cards, and controls render fully within their containers without overlap or cropping." } | 2.29 s |
| banking-ios/defect/layout-defect | 2 | réponse invalide | The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 300, "height": 55 }, "description": "The header text at the top left is cut off by the left edge of the viewport: the brand label reads 'IAYA' and the title reads 'anking' instead of 'Banking', so the first letters are cropped." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 300, "height": 55 }, "description": "The header text at the top left is cut off by the left edge of the viewport: the brand label reads 'IAYA' and the title reads 'anking' instead of 'Banking', so the first letters are cropped." } | 2.89 s |
| banking-ios/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.61 s |
| banking-ios/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.63 s |
| flight-pass-ios/clean/grounding/boarding-pass | 2 | réussite | Cible atteinte | 2.21 s |
| flight-pass-ios/clean/grounding/check-in | 2 | réussite | Cible atteinte | 2.25 s |
| flight-pass-ios/clean/grounding/trips-tab | 2 | réussite | Cible atteinte | 1.87 s |
| flight-pass-ios/clean/grounding/profile-tab | 2 | réussite | Cible atteinte | 2.24 s |
| flight-pass-ios/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; content is fully within the viewport, text fits its containers, and no elements overlap or obscure controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; content is fully within the viewport, text fits its containers, and no elements overlap or obscure controls." } | 2.20 s |
| flight-pass-ios/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 82 }, "description": "A red 'Flight update' banner overlaps the blue check-in button, hiding its label so the button text is cut off and the control is partially obscured." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 82 }, "description": "A red 'Flight update' banner overlaps the blue check-in button, hiding its label so the button text is cut off and the control is partially obscured." } | 2.08 s |
| flight-pass-ios/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.80 s |
| flight-pass-ios/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.71 s |
| smart-home-android/clean/grounding/more | 2 | réussite | Cible atteinte | 1.91 s |
| smart-home-android/clean/grounding/room-search | 2 | réussite | Cible atteinte | 1.75 s |
| smart-home-android/clean/grounding/thermostat-card | 2 | réussite | Cible atteinte | 2.58 s |
| smart-home-android/clean/grounding/add-device | 2 | réussite | Cible atteinte | 1.76 s |
| smart-home-android/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, elements are not overlapping, and nothing is cropped at the viewport edges." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, elements are not overlapping, and nothing is cropped at the viewport edges." } | 1.77 s |
| smart-home-android/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 531, "width": 628, "height": 55 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its chevron is cut off." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 531, "width": 628, "height": 55 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its chevron is cut off." } | 2.13 s |
| smart-home-android/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.74 s |
| smart-home-android/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.58 s |
| commerce-catalog-desktop/clean/grounding/search | 3 | réussite | Cible atteinte | 2.58 s |
| commerce-catalog-desktop/clean/grounding/add-product | 3 | réussite | Cible atteinte | 1.92 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 3 | réussite | Cible atteinte | 3.00 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 3 | réussite | Cible atteinte | 2.02 s |
| commerce-catalog-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact and aligned." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact and aligned." } | 2.56 s |
| commerce-catalog-desktop/defect/layout-defect | 3 | réponse invalide | The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 125, "width": 205, "height": 57 }, "description": "The page heading and eyebrow text are cut off at the left edge of their container. The eyebrow reads 'IANAGEMENT' instead of 'MANAGEMENT' and the title reads 'ducts' instead of 'Products'." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 125, "width": 205, "height": 57 }, "description": "The page heading and eyebrow text are cut off at the left edge of their container. The eyebrow reads 'IANAGEMENT' instead of 'MANAGEMENT' and the title reads 'ducts' instead of 'Products'." } | 3.32 s |
| commerce-catalog-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.68 s |
| commerce-catalog-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.86 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 3 | réussite | Cible atteinte | 2.65 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 3 | réussite | Cible atteinte | 1.85 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 3 | réussite | Cible atteinte | 2.33 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 3 | réussite | Cible atteinte | 2.31 s |
| analytics-dashboard-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, nothing is cropped or overlapping, and controls are unobstructed." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, nothing is cropped or overlapping, and controls are unobstructed." } | 2.16 s |
| analytics-dashboard-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 199, "y": 243, "width": 347, "height": 50 }, "description": "The red heading 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is truncated by the container." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 199, "y": 243, "width": 347, "height": 50 }, "description": "The red heading 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is truncated by the container." } | 2.14 s |
| analytics-dashboard-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.05 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.21 s |
| iot-control-desktop/clean/grounding/devices-nav | 3 | réussite | Cible atteinte | 2.43 s |
| iot-control-desktop/clean/grounding/restart-device | 3 | réussite | Cible atteinte | 2.00 s |
| iot-control-desktop/clean/grounding/auto-mode | 3 | réussite | Cible atteinte | 2.25 s |
| iot-control-desktop/clean/grounding/pump-card | 3 | réussite | Cible atteinte | 1.91 s |
| iot-control-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." } | 2.08 s |
| iot-control-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' banner floats over the table rows, covering part of the Cooling pump and Intake valve rows and breaking the table layout." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' banner floats over the table rows, covering part of the Cooling pump and Intake valve rows and breaking the table layout." } | 2.50 s |
| iot-control-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.88 s |
| iot-control-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.83 s |
| banking-ios/clean/grounding/profile | 3 | réussite | Cible atteinte | 1.75 s |
| banking-ios/clean/grounding/transaction-search | 3 | réussite | Cible atteinte | 2.71 s |
| banking-ios/clean/grounding/hide-balance | 3 | réussite | Cible atteinte | 1.76 s |
| banking-ios/clean/grounding/transfer | 3 | réussite | Cible atteinte | 1.72 s |
| banking-ios/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: text is fully contained, elements are aligned without overlap, and all controls are visible and unobstructed." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: text is fully contained, elements are aligned without overlap, and all controls are visible and unobstructed." } | 1.81 s |
| banking-ios/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 290, "height": 50 }, "description": "The header title 'Banking' and the brand label above it are cut off at the left edge of the screen, showing 'anking' and a partial 'MAYA' label." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 290, "height": 50 }, "description": "The header title 'Banking' and the brand label above it are cut off at the left edge of the screen, showing 'anking' and a partial 'MAYA' label." } | 1.91 s |
| banking-ios/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.70 s |
| banking-ios/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.92 s |
| flight-pass-ios/clean/grounding/boarding-pass | 3 | réussite | Cible atteinte | 2.36 s |
| flight-pass-ios/clean/grounding/check-in | 3 | réussite | Cible atteinte | 2.25 s |
| flight-pass-ios/clean/grounding/trips-tab | 3 | réponse invalide | Cible manquée | 2.21 s |
| flight-pass-ios/clean/grounding/profile-tab | 3 | réussite | Cible atteinte | 1.73 s |
| flight-pass-ios/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: text, cards, button and tab bar are fully visible, aligned and not overlapping or clipped." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: text, cards, button and tab bar are fully visible, aligned and not overlapping or clipped." } | 2.12 s |
| flight-pass-ios/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 80 }, "description": "A red 'Flight update' banner overlaps the blue check-in button, covering most of its label so the button text is cut off and the control is partly hidden." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 80 }, "description": "A red 'Flight update' banner overlaps the blue check-in button, covering most of its label so the button text is cut off and the control is partly hidden." } | 2.08 s |
| flight-pass-ios/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.84 s |
| flight-pass-ios/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.54 s |
| smart-home-android/clean/grounding/more | 3 | réussite | Cible atteinte | 1.58 s |
| smart-home-android/clean/grounding/room-search | 3 | réussite | Cible atteinte | 1.86 s |
| smart-home-android/clean/grounding/thermostat-card | 3 | réussite | Cible atteinte | 2.24 s |
| smart-home-android/clean/grounding/add-device | 3 | réussite | Cible atteinte | 1.75 s |
| smart-home-android/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, elements do not overlap, and controls are fully visible and unobstructed." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, elements do not overlap, and controls are fully visible and unobstructed." } | 1.97 s |
| smart-home-android/defect/layout-defect | 3 | réponse invalide | The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 525, "width": 636, "height": 96 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its right side is cut off." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 525, "width": 636, "height": 96 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its right side is cut off." } | 2.42 s |
| smart-home-android/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.67 s |
| smart-home-android/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.53 s |
Gemini 3.8 Flash · 144 exécutions de cas
Versions de modèle renvoyées : google/gemini-3.8-flash.
Cibles avec le même résultat atteint/manqué sur les trois répétitions : 24 / 24. La concordance peut aussi signifier qu’une cible est manquée à chaque fois.
| Cas | Répétition | Résultat | Issue | Temps |
|---|---|---|---|---|
| commerce-catalog-desktop/clean/grounding/search | 1 | réussite | Cible atteinte | 2.28 s |
| commerce-catalog-desktop/clean/grounding/add-product | 1 | réussite | Cible atteinte | 3.52 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 1 | réussite | Cible atteinte | 2.57 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 1 | réussite | Cible atteinte | 2.31 s |
| commerce-catalog-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 3.65 s |
| commerce-catalog-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 67, "height": 68 }, "description": "The page heading text is clipped on the left edge, cutting off 'Products' into 'ducts' and 'MANAGEMENT' into 'IANAGEMENT'." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 67, "height": 68 }, "description": "The page heading text is clipped on the left edge, cutting off 'Products' into 'ducts' and 'MANAGEMENT' into 'IANAGEMENT'." } | 4.67 s |
| commerce-catalog-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.46 s |
| commerce-catalog-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 3.09 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 1 | réussite | Cible atteinte | 2.08 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 1 | réussite | Cible atteinte | 2.17 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 1 | réussite | Cible atteinte | 2.41 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 1 | réussite | Cible atteinte | 2.25 s |
| analytics-dashboard-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering issues detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering issues detected." } | 6.43 s |
| analytics-dashboard-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 202, "y": 251, "width": 326, "height": 28 }, "description": "The card heading text is clipped at the right boundary of its parent container." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 202, "y": 251, "width": 326, "height": 28 }, "description": "The card heading text is clipped at the right boundary of its parent container." } | 4.58 s |
| analytics-dashboard-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.44 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.84 s |
| iot-control-desktop/clean/grounding/devices-nav | 1 | réussite | Cible atteinte | 2.46 s |
| iot-control-desktop/clean/grounding/restart-device | 1 | réussite | Cible atteinte | 2.57 s |
| iot-control-desktop/clean/grounding/auto-mode | 1 | réussite | Cible atteinte | 2.08 s |
| iot-control-desktop/clean/grounding/pump-card | 1 | réussite | Cible atteinte | 2.26 s |
| iot-control-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering defects detected; elements are well-aligned, legible, and unclipped." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering defects detected; elements are well-aligned, legible, and unclipped." } | 12.60 s |
| iot-control-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The 'Maintenance scheduled' card overlaps the table content and separator lines." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The 'Maintenance scheduled' card overlaps the table content and separator lines." } | 4.28 s |
| iot-control-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.41 s |
| iot-control-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.22 s |
| banking-ios/clean/grounding/profile | 1 | réussite | Cible atteinte | 10.16 s |
| banking-ios/clean/grounding/transaction-search | 1 | réussite | Cible atteinte | 2.53 s |
| banking-ios/clean/grounding/hide-balance | 1 | réussite | Cible atteinte | 1.82 s |
| banking-ios/clean/grounding/transfer | 1 | réussite | Cible atteinte | 2.23 s |
| banking-ios/clean/layout-defect | 1 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 412, "y": 869, "width": 178, "height": 27 }, "description": "Two 'Transfer' text labels are vertically overlapping inside the button, creating a distorted, duplicated text rendering defect." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 412, "y": 869, "width": 178, "height": 27 }, "description": "Two 'Transfer' text labels are vertically overlapping inside the button, creating a distorted, duplicated text rendering defect." } | 13.11 s |
| banking-ios/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 59 }, "description": "The header text ('AYA' / 'banking') is cut off by the left viewport edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 59 }, "description": "The header text ('AYA' / 'banking') is cut off by the left viewport edge." } | 3.90 s |
| banking-ios/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.90 s |
| banking-ios/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.58 s |
| flight-pass-ios/clean/grounding/boarding-pass | 1 | réussite | Cible atteinte | 2.34 s |
| flight-pass-ios/clean/grounding/check-in | 1 | réussite | Cible atteinte | 2.16 s |
| flight-pass-ios/clean/grounding/trips-tab | 1 | réussite | Cible atteinte | 1.65 s |
| flight-pass-ios/clean/grounding/profile-tab | 1 | réussite | Cible atteinte | 1.71 s |
| flight-pass-ios/clean/layout-defect | 1 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 405, "y": 866, "width": 189, "height": 26 }, "description": "The \"Check in\" button text is clipped along its bottom edge inside the button container." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 405, "y": 866, "width": 189, "height": 26 }, "description": "The \"Check in\" button text is clipped along its bottom edge inside the button container." } | 12.77 s |
| flight-pass-ios/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 45, "y": 850, "width": 910, "height": 68 }, "description": "The red 'Flight update' banner overlaps and obscures the blue button beneath it." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 45, "y": 850, "width": 910, "height": 68 }, "description": "The red 'Flight update' banner overlaps and obscures the blue button beneath it." } | 4.16 s |
| flight-pass-ios/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.99 s |
| flight-pass-ios/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.89 s |
| smart-home-android/clean/grounding/more | 1 | réussite | Cible atteinte | 2.05 s |
| smart-home-android/clean/grounding/room-search | 1 | réussite | Cible atteinte | 2.36 s |
| smart-home-android/clean/grounding/thermostat-card | 1 | réussite | Cible atteinte | 2.22 s |
| smart-home-android/clean/grounding/add-device | 1 | réussite | Cible atteinte | 1.55 s |
| smart-home-android/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "None" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "None" } | 8.97 s |
| smart-home-android/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 355, "y": 510, "width": 600, "height": 69 }, "description": "The 'Kitchen lights' card is misaligned and unexpectedly indented/broken compared to the other list items in the 'Quick view' section." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 355, "y": 510, "width": 600, "height": 69 }, "description": "The 'Kitchen lights' card is misaligned and unexpectedly indented/broken compared to the other list items in the 'Quick view' section." } | 7.27 s |
| smart-home-android/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.35 s |
| smart-home-android/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.06 s |
| commerce-catalog-desktop/clean/grounding/search | 2 | réussite | Cible atteinte | 2.53 s |
| commerce-catalog-desktop/clean/grounding/add-product | 2 | réussite | Cible atteinte | 2.38 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 2 | réussite | Cible atteinte | 2.62 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 2 | réussite | Cible atteinte | 2.50 s |
| commerce-catalog-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects detected in the interface." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects detected in the interface." } | 4.11 s |
| commerce-catalog-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 68, "height": 73 }, "description": "The page header text ('Products' and category label above it) is horizontally clipped on the left edge by its container boundary." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 68, "height": 73 }, "description": "The page header text ('Products' and category label above it) is horizontally clipped on the left edge by its container boundary." } | 10.76 s |
| commerce-catalog-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.12 s |
| commerce-catalog-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 3.35 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 2 | réussite | Cible atteinte | 8.75 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 2 | réussite | Cible atteinte | 2.04 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 2 | réussite | Cible atteinte | 1.87 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 2 | réussite | Cible atteinte | 2.24 s |
| analytics-dashboard-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering defects detected." } | 5.95 s |
| analytics-dashboard-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 224, "width": 339, "height": 142 }, "description": "The heading text inside the first KPI card is cut off and clipped by the card's right boundary." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 224, "width": 339, "height": 142 }, "description": "The heading text inside the first KPI card is cut off and clipped by the card's right boundary." } | 3.10 s |
| analytics-dashboard-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.67 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.94 s |
| iot-control-desktop/clean/grounding/devices-nav | 2 | réussite | Cible atteinte | 2.15 s |
| iot-control-desktop/clean/grounding/restart-device | 2 | réussite | Cible atteinte | 2.22 s |
| iot-control-desktop/clean/grounding/auto-mode | 2 | réussite | Cible atteinte | 2.25 s |
| iot-control-desktop/clean/grounding/pump-card | 2 | réussite | Cible atteinte | 2.36 s |
| iot-control-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 185, "y": 118, "width": 227, "height": 70 }, "description": "The subtitle 'FACTORY FLOOR 02' overlaps directly onto the top of the main 'IoT control center' heading." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 185, "y": 118, "width": 227, "height": 70 }, "description": "The subtitle 'FACTORY FLOOR 02' overlaps directly onto the top of the main 'IoT control center' heading." } | 4.70 s |
| iot-control-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' badge or overlay is floating awkwardly and overlapping table row divider lines and table contents." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' badge or overlay is floating awkwardly and overlapping table row divider lines and table contents." } | 2.14 s |
| iot-control-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.26 s |
| iot-control-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.55 s |
| banking-ios/clean/grounding/profile | 2 | réussite | Cible atteinte | 5.36 s |
| banking-ios/clean/grounding/transaction-search | 2 | réussite | Cible atteinte | 2.49 s |
| banking-ios/clean/grounding/hide-balance | 2 | réussite | Cible atteinte | 2.10 s |
| banking-ios/clean/grounding/transfer | 2 | réussite | Cible atteinte | 2.07 s |
| banking-ios/clean/layout-defect | 2 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 413, "y": 868, "width": 174, "height": 22 }, "description": "The 'Transfer' button label has duplicated text overlapping vertically, rendering the text distorted and corrupted." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 413, "y": 868, "width": 174, "height": 22 }, "description": "The 'Transfer' button label has duplicated text overlapping vertically, rendering the text distorted and corrupted." } | 7.07 s |
| banking-ios/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 60 }, "description": "The header text in the upper left corner is clipped off by the left viewport edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 60 }, "description": "The header text in the upper left corner is clipped off by the left viewport edge." } | 4.05 s |
| banking-ios/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.54 s |
| banking-ios/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.07 s |
| flight-pass-ios/clean/grounding/boarding-pass | 2 | réussite | Cible atteinte | 2.06 s |
| flight-pass-ios/clean/grounding/check-in | 2 | réussite | Cible atteinte | 1.79 s |
| flight-pass-ios/clean/grounding/trips-tab | 2 | réussite | Cible atteinte | 2.85 s |
| flight-pass-ios/clean/grounding/profile-tab | 2 | réussite | Cible atteinte | 2.04 s |
| flight-pass-ios/clean/layout-defect | 2 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 852, "width": 908, "height": 59 }, "description": "The 'Check in' label inside the primary action button is vertically clipped at the bottom." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 852, "width": 908, "height": 59 }, "description": "The 'Check in' label inside the primary action button is vertically clipped at the bottom." } | 7.45 s |
| flight-pass-ios/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 46, "y": 849, "width": 908, "height": 67 }, "description": "The red 'Flight update' notification banner overlaps and obscures the blue button behind it." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 46, "y": 849, "width": 908, "height": 67 }, "description": "The red 'Flight update' notification banner overlaps and obscures the blue button behind it." } | 2.35 s |
| flight-pass-ios/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.73 s |
| flight-pass-ios/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.85 s |
| smart-home-android/clean/grounding/more | 2 | réussite | Cible atteinte | 2.12 s |
| smart-home-android/clean/grounding/room-search | 2 | réussite | Cible atteinte | 2.18 s |
| smart-home-android/clean/grounding/thermostat-card | 2 | réussite | Cible atteinte | 2.07 s |
| smart-home-android/clean/grounding/add-device | 2 | réussite | Cible atteinte | 2.02 s |
| smart-home-android/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No rendering defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No rendering defects detected." } | 6.38 s |
| smart-home-android/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 509, "width": 600, "height": 71 }, "description": "The second list card ('Kitchen lights') is severely misaligned horizontally compared to the other list items, breaking the expected vertical list layout." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 509, "width": 600, "height": 71 }, "description": "The second list card ('Kitchen lights') is severely misaligned horizontally compared to the other list items, breaking the expected vertical list layout." } | 8.71 s |
| smart-home-android/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.21 s |
| smart-home-android/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.38 s |
| commerce-catalog-desktop/clean/grounding/search | 3 | réussite | Cible atteinte | 2.32 s |
| commerce-catalog-desktop/clean/grounding/add-product | 3 | réussite | Cible atteinte | 2.40 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 3 | réussite | Cible atteinte | 2.62 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 3 | réussite | Cible atteinte | 3.04 s |
| commerce-catalog-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible defects detected." } | 10.99 s |
| commerce-catalog-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 127, "width": 68, "height": 67 }, "description": "The section title text ('ducts') and subtitle ('IANAGEMENT') are clipped on the left edge by the sidebar/container boundary." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 127, "width": 68, "height": 67 }, "description": "The section title text ('ducts') and subtitle ('IANAGEMENT') are clipped on the left edge by the sidebar/container boundary." } | 2.32 s |
| commerce-catalog-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.51 s |
| commerce-catalog-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.85 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 3 | réussite | Cible atteinte | 9.88 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 3 | réussite | Cible atteinte | 2.26 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 3 | réussite | Cible atteinte | 2.06 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 3 | réussite | Cible atteinte | 2.07 s |
| analytics-dashboard-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual defects detected in the interface." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual defects detected in the interface." } | 5.59 s |
| analytics-dashboard-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 187, "y": 223, "width": 340, "height": 143 }, "description": "The metric card title 'Monthly revenue across all sales chann' is clipped by the container's right boundary." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 187, "y": 223, "width": 340, "height": 143 }, "description": "The metric card title 'Monthly revenue across all sales chann' is clipped by the container's right boundary." } | 5.54 s |
| analytics-dashboard-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.80 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.89 s |
| iot-control-desktop/clean/grounding/devices-nav | 3 | réussite | Cible atteinte | 2.31 s |
| iot-control-desktop/clean/grounding/restart-device | 3 | réussite | Cible atteinte | 2.36 s |
| iot-control-desktop/clean/grounding/auto-mode | 3 | réussite | Cible atteinte | 2.29 s |
| iot-control-desktop/clean/grounding/pump-card | 3 | réussite | Cible atteinte | 2.74 s |
| iot-control-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects detected in the interface." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects detected in the interface." } | 11.74 s |
| iot-control-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 223, "height": 67 }, "description": "The 'Maintenance scheduled' banner overlaps directly on top of the data table and its row borders." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 223, "height": 67 }, "description": "The 'Maintenance scheduled' banner overlaps directly on top of the data table and its row borders." } | 13.83 s |
| iot-control-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.37 s |
| iot-control-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.58 s |
| banking-ios/clean/grounding/profile | 3 | réussite | Cible atteinte | 12.22 s |
| banking-ios/clean/grounding/transaction-search | 3 | réussite | Cible atteinte | 3.27 s |
| banking-ios/clean/grounding/hide-balance | 3 | réussite | Cible atteinte | 2.09 s |
| banking-ios/clean/grounding/transfer | 3 | réussite | Cible atteinte | 1.99 s |
| banking-ios/clean/layout-defect | 3 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 410, "y": 866, "width": 180, "height": 25 }, "description": "Two text labels within the 'Transfer' button overlap directly on top of each other, creating garbled and obscured text." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 410, "y": 866, "width": 180, "height": 25 }, "description": "Two text labels within the 'Transfer' button overlap directly on top of each other, creating garbled and obscured text." } | 8.34 s |
| banking-ios/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 62 }, "description": "The title text 'banking' and the header label above it are cropped by the left edge of the viewport." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 62 }, "description": "The title text 'banking' and the header label above it are cropped by the left edge of the viewport." } | 3.69 s |
| banking-ios/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.23 s |
| banking-ios/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.50 s |
| flight-pass-ios/clean/grounding/boarding-pass | 3 | réussite | Cible atteinte | 2.05 s |
| flight-pass-ios/clean/grounding/check-in | 3 | réussite | Cible atteinte | 2.25 s |
| flight-pass-ios/clean/grounding/trips-tab | 3 | réussite | Cible atteinte | 2.51 s |
| flight-pass-ios/clean/grounding/profile-tab | 3 | réussite | Cible atteinte | 1.92 s |
| flight-pass-ios/clean/layout-defect | 3 | réponse invalide | The answer is not valid JSON. Parsing failed before schema validation could run. JSON parsing error: Unterminated string in JSON at position 213 (line 1 column 214) Malformed answers are not repaired in this pilot and count as incorrect, even when the intended defect category is recognizable. Expected: { "has_issue": false, "category": null, "region": null } Returned: {"has_issue": true, "category": "container_clipping", "region": {"x": 409, "y": 869, "width": 182, "height": 22}, "description": "The bottom portion of the 'Check in' button text is clipped horizontally inside its · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: {"has_issue": true, "category": "container_clipping", "region": {"x": 409, "y": 869, "width": 182, "height": 22}, "description": "The bottom portion of the 'Check in' button text is clipped horizontally inside its | 17.47 s |
| flight-pass-ios/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 66 }, "description": "A red 'Flight update' notification banner overlaps and obscures the blue 'Check in' action button." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 66 }, "description": "A red 'Flight update' notification banner overlaps and obscures the blue 'Check in' action button." } | 4.31 s |
| flight-pass-ios/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.41 s |
| flight-pass-ios/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.49 s |
| smart-home-android/clean/grounding/more | 3 | réussite | Cible atteinte | 1.79 s |
| smart-home-android/clean/grounding/room-search | 3 | réussite | Cible atteinte | 2.07 s |
| smart-home-android/clean/grounding/thermostat-card | 3 | réussite | Cible atteinte | 2.30 s |
| smart-home-android/clean/grounding/add-device | 3 | réussite | Cible atteinte | 2.25 s |
| smart-home-android/clean/layout-defect | 3 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 140, "y": 966, "width": 725, "height": 34 }, "description": "The bottom navigation bar labels ('Home', 'Rooms', 'Activity') are partially cut off by the bottom viewport edge." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 140, "y": 966, "width": 725, "height": 34 }, "description": "The bottom navigation bar labels ('Home', 'Rooms', 'Activity') are partially cut off by the bottom viewport edge." } | 8.35 s |
| smart-home-android/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 355, "y": 510, "width": 600, "height": 70 }, "description": "The 'Kitchen lights' list item card is significantly shifted and misaligned from the vertical grid layout used by surrounding items." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 355, "y": 510, "width": 600, "height": 70 }, "description": "The 'Kitchen lights' list item card is significantly shifted and misaligned from the vertical grid layout used by surrounding items." } | 8.83 s |
| smart-home-android/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.84 s |
| smart-home-android/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.62 s |
Grok 4.7 · 144 exécutions de cas
Versions de modèle renvoyées : x-ai/grok-4.7.
Cibles avec le même résultat atteint/manqué sur les trois répétitions : 16 / 24. La concordance peut aussi signifier qu’une cible est manquée à chaque fois.
| Cas | Répétition | Résultat | Issue | Temps |
|---|---|---|---|---|
| commerce-catalog-desktop/clean/grounding/search | 1 | réussite | Cible atteinte | 12.82 s |
| commerce-catalog-desktop/clean/grounding/add-product | 1 | réussite | Cible atteinte | 7.11 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 1 | réussite | Cible atteinte | 9.00 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 1 | réussite | Cible atteinte | 28.50 s |
| commerce-catalog-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect." } | 12.36 s |
| commerce-catalog-desktop/defect/layout-defect | 1 | réponse invalide | The answer is not valid JSON. Parsing failed before schema validation could run. JSON parsing error: Unterminated string in JSON at position 195 (line 1 column 196) Malformed answers are not repaired in this pilot and count as incorrect, even when the intended defect category is recognizable. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: {"has_issue":true,"category":"container_clipping","region":{"x":278,"y":114,"width":99,"height":65},"description":"The page heading is clipped on the left, showing only “IANAGEMENT” and “ducts.”} · The answer is not valid JSON. Parsing failed before schema validation could run. JSON parsing error: Unterminated string in JSON at position 195 (line 1 column 196) Malformed answers are not repaired in this pilot and count as incorrect, even when the intended defect category is recognizable. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: {"has_issue":true,"category":"container_clipping","region":{"x":278,"y":114,"width":99,"height":65},"description":"The page heading is clipped on the left, showing only “IANAGEMENT” and “ducts.”} | 3.06 s |
| commerce-catalog-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.39 s |
| commerce-catalog-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.97 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 1 | réussite | Cible atteinte | 26.90 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 1 | réussite | Cible atteinte | 9.62 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 1 | réussite | Cible atteinte | 11.49 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 1 | réussite | Cible manquée | 27.51 s |
| analytics-dashboard-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, overflow, overlap, obscured control, or broken reflow is visible." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, overflow, overlap, obscured control, or broken reflow is visible." } | 92.19 s |
| analytics-dashboard-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 204, "y": 226, "width": 318, "height": 34 }, "description": "The revenue card title is truncated mid-word at the card's right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 204, "y": 226, "width": 318, "height": 34 }, "description": "The revenue card title is truncated mid-word at the card's right edge." } | 18.34 s |
| analytics-dashboard-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.54 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 18.39 s |
| iot-control-desktop/clean/grounding/devices-nav | 1 | réussite | Cible atteinte | 13.27 s |
| iot-control-desktop/clean/grounding/restart-device | 1 | réussite | Cible atteinte | 15.28 s |
| iot-control-desktop/clean/grounding/auto-mode | 1 | réussite | Cible atteinte | 25.79 s |
| iot-control-desktop/clean/grounding/pump-card | 1 | réussite | Cible atteinte | 14.44 s |
| iot-control-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect; all text, controls, cards, and table rows are fully contained and aligned." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect; all text, controls, cards, and table rows are fully contained and aligned." } | 6.73 s |
| iot-control-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 556, "width": 219, "height": 66 }, "description": "A red 'Maintenance scheduled' banner overlaps the device table rows." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 556, "width": 219, "height": 66 }, "description": "A red 'Maintenance scheduled' banner overlaps the device table rows." } | 25.49 s |
| iot-control-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.88 s |
| iot-control-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.87 s |
| banking-ios/clean/grounding/profile | 1 | réussite | Cible manquée | 21.93 s |
| banking-ios/clean/grounding/transaction-search | 1 | réussite | Cible atteinte | 10.19 s |
| banking-ios/clean/grounding/hide-balance | 1 | réussite | Cible atteinte | 54.00 s |
| banking-ios/clean/grounding/transfer | 1 | réussite | Cible atteinte | 11.31 s |
| banking-ios/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the listed categories." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the listed categories." } | 8.19 s |
| banking-ios/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 224, "height": 34 }, "description": "The 'Banking' heading is cropped by the left viewport edge, leaving only 'anking' visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 224, "height": 34 }, "description": "The 'Banking' heading is cropped by the left viewport edge, leaving only 'anking' visible." } | 20.63 s |
| banking-ios/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.66 s |
| banking-ios/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.00 s |
| flight-pass-ios/clean/grounding/boarding-pass | 1 | réussite | Cible atteinte | 14.68 s |
| flight-pass-ios/clean/grounding/check-in | 1 | réussite | Cible atteinte | 8.91 s |
| flight-pass-ios/clean/grounding/trips-tab | 1 | réussite | Cible atteinte | 17.82 s |
| flight-pass-ios/clean/grounding/profile-tab | 1 | réussite | Cible atteinte | 17.09 s |
| flight-pass-ios/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 12.00 s |
| flight-pass-ios/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 48, "y": 842, "width": 904, "height": 90 }, "description": "Red Flight update banner overlaps a blue banner, obscuring its text." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 48, "y": 842, "width": 904, "height": 90 }, "description": "Red Flight update banner overlaps a blue banner, obscuring its text." } | 19.37 s |
| flight-pass-ios/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.86 s |
| flight-pass-ios/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.58 s |
| smart-home-android/clean/grounding/more | 1 | réussite | Cible manquée | 8.75 s |
| smart-home-android/clean/grounding/room-search | 1 | réussite | Cible atteinte | 9.33 s |
| smart-home-android/clean/grounding/thermostat-card | 1 | réussite | Cible atteinte | 13.72 s |
| smart-home-android/clean/grounding/add-device | 1 | réussite | Cible manquée | 8.84 s |
| smart-home-android/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 3.17 s |
| smart-home-android/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 366, "y": 509, "width": 588, "height": 83 }, "description": "The Kitchen lights card is shifted right and does not align with the full-width cards above and below it." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 366, "y": 509, "width": 588, "height": 83 }, "description": "The Kitchen lights card is shifted right and does not align with the full-width cards above and below it." } | 20.28 s |
| smart-home-android/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.88 s |
| smart-home-android/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.52 s |
| commerce-catalog-desktop/clean/grounding/search | 2 | réussite | Cible atteinte | 26.80 s |
| commerce-catalog-desktop/clean/grounding/add-product | 2 | réussite | Cible atteinte | 7.36 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 2 | réussite | Cible atteinte | 20.98 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 2 | réussite | Cible manquée | 17.11 s |
| commerce-catalog-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 5.63 s |
| commerce-catalog-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 276, "y": 112, "width": 103, "height": 68 }, "description": "The page heading is clipped at its left edge, cutting off the beginning of both heading lines." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 276, "y": 112, "width": 103, "height": 68 }, "description": "The page heading is clipped at its left edge, cutting off the beginning of both heading lines." } | 3.07 s |
| commerce-catalog-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.73 s |
| commerce-catalog-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.08 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 2 | réussite | Cible atteinte | 18.38 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 2 | réussite | Cible atteinte | 16.61 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 2 | réussite | Cible manquée | 38.77 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 2 | réussite | Cible manquée | 21.24 s |
| analytics-dashboard-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect." } | 45.43 s |
| analytics-dashboard-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 228, "width": 324, "height": 46 }, "description": "The metric card heading is cut off mid-word at the card's right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 228, "width": 324, "height": 46 }, "description": "The metric card heading is cut off mid-word at the card's right edge." } | 14.96 s |
| analytics-dashboard-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.22 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.97 s |
| iot-control-desktop/clean/grounding/devices-nav | 2 | réussite | Cible atteinte | 21.10 s |
| iot-control-desktop/clean/grounding/restart-device | 2 | réussite | Cible atteinte | 11.53 s |
| iot-control-desktop/clean/grounding/auto-mode | 2 | réussite | Cible manquée | 16.62 s |
| iot-control-desktop/clean/grounding/pump-card | 2 | réussite | Cible atteinte | 22.31 s |
| iot-control-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects." } | 13.76 s |
| iot-control-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 458, "y": 555, "width": 224, "height": 69 }, "description": "A red \"Maintenance scheduled\" banner overlaps and obscures the device table rows." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 458, "y": 555, "width": 224, "height": 69 }, "description": "A red \"Maintenance scheduled\" banner overlaps and obscures the device table rows." } | 12.82 s |
| iot-control-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.78 s |
| iot-control-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.89 s |
| banking-ios/clean/grounding/profile | 2 | réussite | Cible manquée | 20.19 s |
| banking-ios/clean/grounding/transaction-search | 2 | réussite | Cible atteinte | 24.96 s |
| banking-ios/clean/grounding/hide-balance | 2 | réussite | Cible manquée | 8.56 s |
| banking-ios/clean/grounding/transfer | 2 | réussite | Cible atteinte | 8.18 s |
| banking-ios/clean/layout-defect | 2 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 58, "y": 962, "width": 814, "height": 38 }, "description": "Bottom navigation labels are cut off by the lower viewport edge." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 58, "y": 962, "width": 814, "height": 38 }, "description": "Bottom navigation labels are cut off by the lower viewport edge." } | 18.85 s |
| banking-ios/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 58, "width": 500, "height": 48 }, "description": "The 'Banking' heading is cropped at the left viewport edge, with the leading 'B' cut off." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 58, "width": 500, "height": 48 }, "description": "The 'Banking' heading is cropped at the left viewport edge, with the leading 'B' cut off." } | 20.32 s |
| banking-ios/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.54 s |
| banking-ios/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.83 s |
| flight-pass-ios/clean/grounding/boarding-pass | 2 | réussite | Cible atteinte | 12.05 s |
| flight-pass-ios/clean/grounding/check-in | 2 | réussite | Cible manquée | 15.24 s |
| flight-pass-ios/clean/grounding/trips-tab | 2 | réussite | Cible atteinte | 23.46 s |
| flight-pass-ios/clean/grounding/profile-tab | 2 | réussite | Cible atteinte | 25.79 s |
| flight-pass-ios/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect; content fits the viewport without clipping, overflow, overlap, or broken reflow." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect; content fits the viewport without clipping, overflow, overlap, or broken reflow." } | 7.64 s |
| flight-pass-ios/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 42, "y": 812, "width": 918, "height": 92 }, "description": "A red flight-update banner overlaps a blue card behind it, obscuring that card's text." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 42, "y": 812, "width": 918, "height": 92 }, "description": "A red flight-update banner overlaps a blue card behind it, obscuring that card's text." } | 12.45 s |
| flight-pass-ios/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.96 s |
| flight-pass-ios/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.77 s |
| smart-home-android/clean/grounding/more | 2 | réussite | Cible manquée | 16.25 s |
| smart-home-android/clean/grounding/room-search | 2 | réussite | Cible atteinte | 16.47 s |
| smart-home-android/clean/grounding/thermostat-card | 2 | réussite | Cible atteinte | 19.16 s |
| smart-home-android/clean/grounding/add-device | 2 | réussite | Cible atteinte | 9.97 s |
| smart-home-android/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect." } | 15.40 s |
| smart-home-android/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 328, "y": 567, "width": 632, "height": 110 }, "description": "The Kitchen lights card is shifted right and overlaps the Front door card beneath it." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 328, "y": 567, "width": 632, "height": 110 }, "description": "The Kitchen lights card is shifted right and overlaps the Front door card beneath it." } | 40.90 s |
| smart-home-android/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.51 s |
| smart-home-android/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 5.93 s |
| commerce-catalog-desktop/clean/grounding/search | 3 | réussite | Cible atteinte | 27.72 s |
| commerce-catalog-desktop/clean/grounding/add-product | 3 | réponse invalide | Cible manquée | 11.21 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 3 | réussite | Cible atteinte | 21.57 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 3 | réussite | Cible manquée | 17.41 s |
| commerce-catalog-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, overflow, overlap, obscured control, or broken reflow is visible." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, overflow, overlap, obscured control, or broken reflow is visible." } | 13.76 s |
| commerce-catalog-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 273, "y": 116, "width": 102, "height": 62 }, "description": "Heading text is clipped at its left edge, showing incomplete words." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 273, "y": 116, "width": 102, "height": 62 }, "description": "Heading text is clipped at its left edge, showing incomplete words." } | 3.93 s |
| commerce-catalog-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.19 s |
| commerce-catalog-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.49 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 3 | réussite | Cible atteinte | 24.18 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 3 | réussite | Cible atteinte | 9.67 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 3 | réussite | Cible atteinte | 6.95 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 3 | réussite | Cible manquée | 10.42 s |
| analytics-dashboard-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the listed categories." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the listed categories." } | 35.63 s |
| analytics-dashboard-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 294, "y": 224, "width": 459, "height": 34 }, "description": "The revenue card heading is cut off at the card's right edge, truncating the title text." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 294, "y": 224, "width": 459, "height": 34 }, "description": "The revenue card heading is cut off at the card's right edge, truncating the title text." } | 18.82 s |
| analytics-dashboard-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 6.03 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.56 s |
| iot-control-desktop/clean/grounding/devices-nav | 3 | réussite | Cible atteinte | 13.95 s |
| iot-control-desktop/clean/grounding/restart-device | 3 | réussite | Cible atteinte | 8.91 s |
| iot-control-desktop/clean/grounding/auto-mode | 3 | réussite | Cible atteinte | 18.87 s |
| iot-control-desktop/clean/grounding/pump-card | 3 | réussite | Cible atteinte | 19.79 s |
| iot-control-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect; text, cards, table, and controls render within their containers." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect; text, cards, table, and controls render within their containers." } | 13.84 s |
| iot-control-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 463, "y": 556, "width": 223, "height": 68 }, "description": "A red maintenance banner overlaps the device table, covering the Cooling pump status and row boundary." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 463, "y": 556, "width": 223, "height": 68 }, "description": "A red maintenance banner overlaps the device table, covering the Cooling pump status and row boundary." } | 5.51 s |
| iot-control-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.98 s |
| iot-control-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.08 s |
| banking-ios/clean/grounding/profile | 3 | réussite | Cible manquée | 17.73 s |
| banking-ios/clean/grounding/transaction-search | 3 | réussite | Cible atteinte | 17.27 s |
| banking-ios/clean/grounding/hide-balance | 3 | réussite | Cible manquée | 21.90 s |
| banking-ios/clean/grounding/transfer | 3 | réussite | Cible atteinte | 7.91 s |
| banking-ios/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 9.43 s |
| banking-ios/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 49, "width": 162, "height": 44 }, "description": "The Banking header title is cropped at the left viewport edge, leaving only anking visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 49, "width": 162, "height": 44 }, "description": "The Banking header title is cropped at the left viewport edge, leaving only anking visible." } | 12.40 s |
| banking-ios/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.41 s |
| banking-ios/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.02 s |
| flight-pass-ios/clean/grounding/boarding-pass | 3 | réussite | Cible atteinte | 12.12 s |
| flight-pass-ios/clean/grounding/check-in | 3 | réussite | Cible manquée | 17.61 s |
| flight-pass-ios/clean/grounding/trips-tab | 3 | réussite | Cible atteinte | 18.95 s |
| flight-pass-ios/clean/grounding/profile-tab | 3 | réussite | Cible atteinte | 32.81 s |
| flight-pass-ios/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect in the specified categories." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect in the specified categories." } | 10.62 s |
| flight-pass-ios/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 41, "y": 829, "width": 877, "height": 90 }, "description": "The red Flight update banner overlaps the blue banner behind it, obscuring and clipping its text." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 41, "y": 829, "width": 877, "height": 90 }, "description": "The red Flight update banner overlaps the blue banner behind it, obscuring and clipping its text." } | 17.93 s |
| flight-pass-ios/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.46 s |
| flight-pass-ios/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.66 s |
| smart-home-android/clean/grounding/more | 3 | réussite | Cible manquée | 4.33 s |
| smart-home-android/clean/grounding/room-search | 3 | réussite | Cible atteinte | 6.48 s |
| smart-home-android/clean/grounding/thermostat-card | 3 | réussite | Cible manquée | 22.65 s |
| smart-home-android/clean/grounding/add-device | 3 | réussite | Cible atteinte | 9.77 s |
| smart-home-android/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the specified categories." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the specified categories." } | 10.69 s |
| smart-home-android/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 430, "y": 510, "width": 528, "height": 80 }, "description": "The Kitchen lights card is indented and narrower than the full-width sibling cards, breaking the list reflow." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 430, "y": 510, "width": 528, "height": 80 }, "description": "The Kitchen lights card is indented and narrower than the full-width sibling cards, breaking the list reflow." } | 23.64 s |
| smart-home-android/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.57 s |
| smart-home-android/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.55 s |
Qwen3.8 Max (0902) · 144 exécutions de cas
Versions de modèle renvoyées : qwen/qwen3.8-max-0902.
Cibles avec le même résultat atteint/manqué sur les trois répétitions : 24 / 24. La concordance peut aussi signifier qu’une cible est manquée à chaque fois.
| Cas | Répétition | Résultat | Issue | Temps |
|---|---|---|---|---|
| commerce-catalog-desktop/clean/grounding/search | 1 | réussite | Cible atteinte | 6.70 s |
| commerce-catalog-desktop/clean/grounding/add-product | 1 | réussite | Cible atteinte | 9.03 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 1 | réussite | Cible atteinte | 9.96 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 1 | réussite | Cible atteinte | 7.91 s |
| commerce-catalog-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } | 11.01 s |
| commerce-catalog-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 190, "y": 120, "width": 180, "height": 75 }, "description": "Page heading is cut off at the left edge, showing 'ducts' instead of 'Products' and 'IANAGEMENT' instead of 'MANAGEMENT'." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 190, "y": 120, "width": 180, "height": 75 }, "description": "Page heading is cut off at the left edge, showing 'ducts' instead of 'Products' and 'IANAGEMENT' instead of 'MANAGEMENT'." } | 15.27 s |
| commerce-catalog-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 4.24 s |
| commerce-catalog-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 5.16 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 1 | réussite | Cible atteinte | 14.38 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 1 | réussite | Cible atteinte | 11.24 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 1 | réussite | Cible atteinte | 10.90 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 1 | réussite | Cible atteinte | 11.42 s |
| analytics-dashboard-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } | 11.64 s |
| analytics-dashboard-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 200, "y": 245, "width": 330, "height": 40 }, "description": "Card title 'Monthly revenue across all sales channels' is cut off at the card's right edge, truncating the word 'channels'." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 200, "y": 245, "width": 330, "height": 40 }, "description": "Card title 'Monthly revenue across all sales channels' is cut off at the card's right edge, truncating the word 'channels'." } | 14.56 s |
| analytics-dashboard-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 4.22 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 5.52 s |
| iot-control-desktop/clean/grounding/devices-nav | 1 | réussite | Cible atteinte | 11.78 s |
| iot-control-desktop/clean/grounding/restart-device | 1 | réussite | Cible atteinte | 11.50 s |
| iot-control-desktop/clean/grounding/auto-mode | 1 | réussite | Cible atteinte | 14.37 s |
| iot-control-desktop/clean/grounding/pump-card | 1 | réussite | Cible atteinte | 8.43 s |
| iot-control-desktop/clean/layout-defect | 1 | réponse invalide | The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: cards, table, toggle, and button render fully within their containers without cropping, clipping, overflow, overlap, or obscured controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: cards, table, toggle, and button render fully within their containers without cropping, clipping, overflow, overlap, or obscured controls." } | 10.94 s |
| iot-control-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; toast notification overlays content intentionally without obscuring actionable controls." } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; toast notification overlays content intentionally without obscuring actionable controls." } | 14.07 s |
| iot-control-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.83 s |
| iot-control-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 6.18 s |
| banking-ios/clean/grounding/profile | 1 | réussite | Cible atteinte | 9.52 s |
| banking-ios/clean/grounding/transaction-search | 1 | réussite | Cible atteinte | 6.30 s |
| banking-ios/clean/grounding/hide-balance | 1 | réussite | Cible atteinte | 18.69 s |
| banking-ios/clean/grounding/transfer | 1 | réussite | Cible atteinte | 11.96 s |
| banking-ios/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all containers, text, and controls render fully within the viewport without clipping, overflow, or overlap." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all containers, text, and controls render fully within the viewport without clipping, overflow, or overlap." } | 9.44 s |
| banking-ios/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 280, "height": 60 }, "description": "Page heading 'Banking' is clipped at the left edge; the leading 'B' is cut off by the viewport." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 280, "height": 60 }, "description": "Page heading 'Banking' is clipped at the left edge; the leading 'B' is cut off by the viewport." } | 9.56 s |
| banking-ios/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.62 s |
| banking-ios/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 4.11 s |
| flight-pass-ios/clean/grounding/boarding-pass | 1 | réussite | Cible atteinte | 8.08 s |
| flight-pass-ios/clean/grounding/check-in | 1 | réussite | Cible atteinte | 6.42 s |
| flight-pass-ios/clean/grounding/trips-tab | 1 | réussite | Cible atteinte | 8.58 s |
| flight-pass-ios/clean/grounding/profile-tab | 1 | réussite | Cible atteinte | 7.65 s |
| flight-pass-ios/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } | 8.78 s |
| flight-pass-ios/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 855, "width": 920, "height": 65 }, "description": "Red 'Flight update' toast banner overlaps and obscures the blue 'Check in' button text beneath it." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 855, "width": 920, "height": 65 }, "description": "Red 'Flight update' toast banner overlaps and obscures the blue 'Check in' button text beneath it." } | 14.78 s |
| flight-pass-ios/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.73 s |
| flight-pass-ios/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 4.51 s |
| smart-home-android/clean/grounding/more | 1 | réussite | Cible atteinte | 9.39 s |
| smart-home-android/clean/grounding/room-search | 1 | réussite | Cible atteinte | 5.70 s |
| smart-home-android/clean/grounding/thermostat-card | 1 | réussite | Cible atteinte | 7.16 s |
| smart-home-android/clean/grounding/add-device | 1 | réussite | Cible atteinte | 6.43 s |
| smart-home-android/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without overlap or clipping." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without overlap or clipping." } | 9.54 s |
| smart-home-android/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 355, "y": 510, "width": 645, "height": 70 }, "description": "The 'Kitchen lights' card (item 2) is shifted right and clipped at the right viewport edge, cutting off its border and chevron control." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 355, "y": 510, "width": 645, "height": 70 }, "description": "The 'Kitchen lights' card (item 2) is shifted right and clipped at the right viewport edge, cutting off its border and chevron control." } | 14.85 s |
| smart-home-android/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.91 s |
| smart-home-android/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 4.46 s |
| commerce-catalog-desktop/clean/grounding/search | 2 | réussite | Cible atteinte | 7.87 s |
| commerce-catalog-desktop/clean/grounding/add-product | 2 | réussite | Cible atteinte | 10.41 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 2 | réussite | Cible atteinte | 10.06 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 2 | réussite | Cible atteinte | 12.58 s |
| commerce-catalog-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all content fits within containers and viewport." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all content fits within containers and viewport." } | 8.64 s |
| commerce-catalog-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 190, "y": 120, "width": 180, "height": 75 }, "description": "Page heading 'Products' is clipped on the left, showing 'ducts', and the 'MANAGEMENT' eyebrow label is cut off at the container edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 190, "y": 120, "width": 180, "height": 75 }, "description": "Page heading 'Products' is clipped on the left, showing 'ducts', and the 'MANAGEMENT' eyebrow label is cut off at the container edge." } | 13.24 s |
| commerce-catalog-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.67 s |
| commerce-catalog-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 5.86 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 2 | réussite | Cible atteinte | 11.26 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 2 | réussite | Cible atteinte | 9.96 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 2 | réussite | Cible atteinte | 9.24 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 2 | réussite | Cible atteinte | 12.36 s |
| analytics-dashboard-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all content fits within containers and viewport." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all content fits within containers and viewport." } | 10.56 s |
| analytics-dashboard-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 245, "width": 340, "height": 40 }, "description": "Card title 'Monthly revenue across all sales channels' is truncated at the right edge of its container, cutting off the final letters." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 245, "width": 340, "height": 40 }, "description": "Card title 'Monthly revenue across all sales channels' is truncated at the right edge of its container, cutting off the final letters." } | 13.44 s |
| analytics-dashboard-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 4.18 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 5.57 s |
| iot-control-desktop/clean/grounding/devices-nav | 2 | réussite | Cible atteinte | 6.57 s |
| iot-control-desktop/clean/grounding/restart-device | 2 | réussite | Cible atteinte | 8.10 s |
| iot-control-desktop/clean/grounding/auto-mode | 2 | réussite | Cible atteinte | 10.17 s |
| iot-control-desktop/clean/grounding/pump-card | 2 | réussite | Cible atteinte | 5.96 s |
| iot-control-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without overlap or clipping." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without overlap or clipping." } | 9.94 s |
| iot-control-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 223, "height": 63 }, "description": "A red 'Maintenance scheduled' toast/banner overlays the device table rows, covering the area between the Cooling pump and Intake valve entries." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 223, "height": 63 }, "description": "A red 'Maintenance scheduled' toast/banner overlays the device table rows, covering the area between the Cooling pump and Intake valve entries." } | 22.10 s |
| iot-control-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.75 s |
| iot-control-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 5.03 s |
| banking-ios/clean/grounding/profile | 2 | réussite | Cible atteinte | 8.60 s |
| banking-ios/clean/grounding/transaction-search | 2 | réussite | Cible atteinte | 8.13 s |
| banking-ios/clean/grounding/hide-balance | 2 | réussite | Cible atteinte | 6.74 s |
| banking-ios/clean/grounding/transfer | 2 | réussite | Cible atteinte | 6.15 s |
| banking-ios/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } | 8.60 s |
| banking-ios/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 300, "height": 60 }, "description": "Page title 'Banking' is cut off at the left viewport edge, showing only 'anking'." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 300, "height": 60 }, "description": "Page title 'Banking' is cut off at the left viewport edge, showing only 'anking'." } | 11.66 s |
| banking-ios/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.97 s |
| banking-ios/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 6.69 s |
| flight-pass-ios/clean/grounding/boarding-pass | 2 | réussite | Cible atteinte | 10.94 s |
| flight-pass-ios/clean/grounding/check-in | 2 | réussite | Cible atteinte | 6.76 s |
| flight-pass-ios/clean/grounding/trips-tab | 2 | réussite | Cible atteinte | 7.43 s |
| flight-pass-ios/clean/grounding/profile-tab | 2 | réussite | Cible atteinte | 7.58 s |
| flight-pass-ios/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all text fits, controls are unobscured, and elements are properly spaced." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all text fits, controls are unobscured, and elements are properly spaced." } | 9.35 s |
| flight-pass-ios/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 920, "height": 70 }, "description": "The red 'Flight update' toast banner overlaps and obscures the blue 'Check in' button text beneath it." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 920, "height": 70 }, "description": "The red 'Flight update' toast banner overlaps and obscures the blue 'Check in' button text beneath it." } | 11.78 s |
| flight-pass-ios/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.27 s |
| flight-pass-ios/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 5.39 s |
| smart-home-android/clean/grounding/more | 2 | réussite | Cible atteinte | 6.06 s |
| smart-home-android/clean/grounding/room-search | 2 | réussite | Cible atteinte | 6.43 s |
| smart-home-android/clean/grounding/thermostat-card | 2 | réussite | Cible atteinte | 6.58 s |
| smart-home-android/clean/grounding/add-device | 2 | réussite | Cible atteinte | 6.15 s |
| smart-home-android/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without cropping, clipping, overflow, overlap, or obscured controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without cropping, clipping, overflow, overlap, or obscured controls." } | 9.39 s |
| smart-home-android/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 350, "y": 510, "width": 650, "height": 75 }, "description": "The 'Kitchen lights' quick-view card is offset to the right and clipped by the right viewport edge, breaking the list's alignment." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 350, "y": 510, "width": 650, "height": 75 }, "description": "The 'Kitchen lights' quick-view card is offset to the right and clipped by the right viewport edge, breaking the list's alignment." } | 17.60 s |
| smart-home-android/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 4.70 s |
| smart-home-android/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 6.14 s |
| commerce-catalog-desktop/clean/grounding/search | 3 | réussite | Cible atteinte | 9.29 s |
| commerce-catalog-desktop/clean/grounding/add-product | 3 | réussite | Cible atteinte | 10.12 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 3 | réussite | Cible atteinte | 9.57 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 3 | réussite | Cible atteinte | 5.48 s |
| commerce-catalog-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within their containers and the viewport." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within their containers and the viewport." } | 7.69 s |
| commerce-catalog-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 165, "y": 110, "width": 120, "height": 70 }, "description": "Page heading is cut off on the left: 'MANAGEMENT' appears as 'IANAGEMENT' and 'Products' as 'ducts', clipped by the sidebar edge." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 165, "y": 110, "width": 120, "height": 70 }, "description": "Page heading is cut off on the left: 'MANAGEMENT' appears as 'IANAGEMENT' and 'Products' as 'ducts', clipped by the sidebar edge." } | 16.10 s |
| commerce-catalog-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 4.78 s |
| commerce-catalog-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 8.14 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 3 | réussite | Cible atteinte | 7.93 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 3 | réussite | Cible atteinte | 12.24 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 3 | réussite | Cible atteinte | 15.80 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 3 | réussite | Cible atteinte | 11.01 s |
| analytics-dashboard-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } | 10.27 s |
| analytics-dashboard-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 248, "width": 328, "height": 32 }, "description": "Card title \"Monthly revenue across all sales chann\" is truncated at the card's right edge, cutting off the word \"channels\"." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 248, "width": 328, "height": 32 }, "description": "Card title \"Monthly revenue across all sales chann\" is truncated at the card's right edge, cutting off the word \"channels\"." } | 12.46 s |
| analytics-dashboard-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 7.10 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 4.95 s |
| iot-control-desktop/clean/grounding/devices-nav | 3 | réussite | Cible atteinte | 5.90 s |
| iot-control-desktop/clean/grounding/restart-device | 3 | réussite | Cible atteinte | 8.96 s |
| iot-control-desktop/clean/grounding/auto-mode | 3 | réussite | Cible atteinte | 7.81 s |
| iot-control-desktop/clean/grounding/pump-card | 3 | réussite | Cible atteinte | 8.01 s |
| iot-control-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects; layout is intact with no cropping, clipping, overflow, overlap, or obscured controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects; layout is intact with no cropping, clipping, overflow, overlap, or obscured controls." } | 10.18 s |
| iot-control-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 222, "height": 65 }, "description": "A red 'Maintenance scheduled' toast/banner overlaps the device table rows, covering content between the Cooling pump and Intake valve rows." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 222, "height": 65 }, "description": "A red 'Maintenance scheduled' toast/banner overlaps the device table rows, covering content between the Cooling pump and Intake valve rows." } | 11.95 s |
| iot-control-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.77 s |
| iot-control-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 6.03 s |
| banking-ios/clean/grounding/profile | 3 | réussite | Cible atteinte | 9.36 s |
| banking-ios/clean/grounding/transaction-search | 3 | réussite | Cible atteinte | 6.13 s |
| banking-ios/clean/grounding/hide-balance | 3 | réussite | Cible atteinte | 8.05 s |
| banking-ios/clean/grounding/transfer | 3 | réussite | Cible atteinte | 6.87 s |
| banking-ios/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without cropping, clipping, overflow, overlap, or obscured controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without cropping, clipping, overflow, overlap, or obscured controls." } | 9.48 s |
| banking-ios/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 280, "height": 50 }, "description": "Page title 'Banking' is cut off at the left viewport edge, showing only 'anking'." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 280, "height": 50 }, "description": "Page title 'Banking' is cut off at the left viewport edge, showing only 'anking'." } | 9.28 s |
| banking-ios/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 4.25 s |
| banking-ios/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 4.52 s |
| flight-pass-ios/clean/grounding/boarding-pass | 3 | réussite | Cible atteinte | 7.59 s |
| flight-pass-ios/clean/grounding/check-in | 3 | réussite | Cible atteinte | 9.32 s |
| flight-pass-ios/clean/grounding/trips-tab | 3 | réussite | Cible atteinte | 6.66 s |
| flight-pass-ios/clean/grounding/profile-tab | 3 | réussite | Cible atteinte | 8.68 s |
| flight-pass-ios/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all text fits within containers, controls are unobscured, and elements do not overlap or clip." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all text fits within containers, controls are unobscured, and elements do not overlap or clip." } | 8.34 s |
| flight-pass-ios/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 45, "y": 850, "width": 910, "height": 70 }, "description": "A red 'Flight update' toast banner overlaps and obscures a blue action button ('Check in') near the bottom of the screen." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 45, "y": 850, "width": 910, "height": 70 }, "description": "A red 'Flight update' toast banner overlaps and obscures a blue action button ('Check in') near the bottom of the screen." } | 14.65 s |
| flight-pass-ios/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.98 s |
| flight-pass-ios/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 4.74 s |
| smart-home-android/clean/grounding/more | 3 | réussite | Cible atteinte | 8.92 s |
| smart-home-android/clean/grounding/room-search | 3 | réussite | Cible atteinte | 6.79 s |
| smart-home-android/clean/grounding/thermostat-card | 3 | réussite | Cible atteinte | 7.32 s |
| smart-home-android/clean/grounding/add-device | 3 | réussite | Cible atteinte | 6.50 s |
| smart-home-android/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } | 8.80 s |
| smart-home-android/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 355, "y": 508, "width": 645, "height": 70 }, "description": "The 'Kitchen lights' quick-view card is offset to the right and its right edge is clipped by the viewport, unlike the other aligned cards." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 355, "y": 508, "width": 645, "height": 70 }, "description": "The 'Kitchen lights' quick-view card is offset to the right and its right edge is clipped by the viewport, unlike the other aligned cards." } | 14.15 s |
| smart-home-android/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.64 s |
| smart-home-android/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 5.56 s |
Muse Spark 1.3 · 144 exécutions de cas
Versions de modèle renvoyées : meta/muse-spark-1.3.
Cibles avec le même résultat atteint/manqué sur les trois répétitions : 23 / 24. La concordance peut aussi signifier qu’une cible est manquée à chaque fois.
| Cas | Répétition | Résultat | Issue | Temps |
|---|---|---|---|---|
| commerce-catalog-desktop/clean/grounding/search | 1 | réussite | Cible atteinte | 5.13 s |
| commerce-catalog-desktop/clean/grounding/add-product | 1 | réussite | Cible atteinte | 2.39 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 1 | réussite | Cible atteinte | 3.01 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 1 | réussite | Cible atteinte | 6.57 s |
| commerce-catalog-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defects observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defects observed.", "has_issue": false, "region": null } | 4.09 s |
| commerce-catalog-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Product heading clipped on left edge showing 'IANAGEMENT' and 'ducts' instead of full title.", "has_issue": true, "region": { "height": 65, "width": 68, "x": 191, "y": 126 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Product heading clipped on left edge showing 'IANAGEMENT' and 'ducts' instead of full title.", "has_issue": true, "region": { "height": 65, "width": 68, "x": 191, "y": 126 } } | 7.80 s |
| commerce-catalog-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.66 s |
| commerce-catalog-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.79 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 1 | réussite | Cible atteinte | 4.96 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 1 | réussite | Cible atteinte | 5.08 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 1 | réussite | Cible atteinte | 3.09 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 1 | réussite | Cible atteinte | 4.94 s |
| analytics-dashboard-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } | 4.07 s |
| analytics-dashboard-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card title truncated to 'Monthly revenue across all sales chann', cutting off end of word.", "has_issue": true, "region": { "height": 30, "width": 330, "x": 200, "y": 248 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card title truncated to 'Monthly revenue across all sales chann', cutting off end of word.", "has_issue": true, "region": { "height": 30, "width": 330, "x": 200, "y": 248 } } | 8.22 s |
| analytics-dashboard-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.36 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.71 s |
| iot-control-desktop/clean/grounding/devices-nav | 1 | réussite | Cible atteinte | 3.78 s |
| iot-control-desktop/clean/grounding/restart-device | 1 | réussite | Cible atteinte | 3.54 s |
| iot-control-desktop/clean/grounding/auto-mode | 1 | réussite | Cible atteinte | 6.48 s |
| iot-control-desktop/clean/grounding/pump-card | 1 | réussite | Cible atteinte | 5.57 s |
| iot-control-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } | 8.45 s |
| iot-control-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } | 5.07 s |
| iot-control-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.95 s |
| iot-control-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.71 s |
| banking-ios/clean/grounding/profile | 1 | réussite | Cible atteinte | 6.58 s |
| banking-ios/clean/grounding/transaction-search | 1 | réussite | Cible atteinte | 3.56 s |
| banking-ios/clean/grounding/hide-balance | 1 | réussite | Cible manquée | 8.77 s |
| banking-ios/clean/grounding/transfer | 1 | réussite | Cible atteinte | 3.70 s |
| banking-ios/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } | 4.67 s |
| banking-ios/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "Header title 'banking' is cropped at left viewport edge with first letter partially cut off.", "has_issue": true, "region": { "height": 60, "width": 260, "x": 35, "y": 58 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "Header title 'banking' is cropped at left viewport edge with first letter partially cut off.", "has_issue": true, "region": { "height": 60, "width": 260, "x": 35, "y": 58 } } | 5.79 s |
| banking-ios/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.00 s |
| banking-ios/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.72 s |
| flight-pass-ios/clean/grounding/boarding-pass | 1 | réussite | Cible manquée | 4.66 s |
| flight-pass-ios/clean/grounding/check-in | 1 | réussite | Cible atteinte | 3.46 s |
| flight-pass-ios/clean/grounding/trips-tab | 1 | réussite | Cible atteinte | 5.15 s |
| flight-pass-ios/clean/grounding/profile-tab | 1 | réussite | Cible atteinte | 4.18 s |
| flight-pass-ios/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } | 4.41 s |
| flight-pass-ios/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red Flight update banner overlaps blue banner beneath it, obscuring its text.", "has_issue": true, "region": { "height": 62, "width": 912, "x": 43, "y": 863 } } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red Flight update banner overlaps blue banner beneath it, obscuring its text.", "has_issue": true, "region": { "height": 62, "width": 912, "x": 43, "y": 863 } } | 9.53 s |
| flight-pass-ios/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.16 s |
| flight-pass-ios/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.53 s |
| smart-home-android/clean/grounding/more | 1 | réussite | Cible atteinte | 4.07 s |
| smart-home-android/clean/grounding/room-search | 1 | réussite | Cible atteinte | 5.98 s |
| smart-home-android/clean/grounding/thermostat-card | 1 | réussite | Cible atteinte | 4.82 s |
| smart-home-android/clean/grounding/add-device | 1 | réussite | Cible atteinte | 3.71 s |
| smart-home-android/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } | 4.54 s |
| smart-home-android/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": "viewport_cropping", "description": "Kitchen lights Quick view card is offset right and clipped by viewport edge.", "has_issue": true, "region": { "height": 116, "width": 845, "x": 155, "y": 534 } } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": "viewport_cropping", "description": "Kitchen lights Quick view card is offset right and clipped by viewport edge.", "has_issue": true, "region": { "height": 116, "width": 845, "x": 155, "y": 534 } } | 8.58 s |
| smart-home-android/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.20 s |
| smart-home-android/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 3.74 s |
| commerce-catalog-desktop/clean/grounding/search | 2 | réussite | Cible atteinte | 4.15 s |
| commerce-catalog-desktop/clean/grounding/add-product | 2 | réussite | Cible atteinte | 3.72 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 2 | réussite | Cible atteinte | 2.27 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 2 | réussite | Cible atteinte | 5.52 s |
| commerce-catalog-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } | 4.06 s |
| commerce-catalog-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Main heading clipped on left showing 'ducts' and 'IANAGEMENT' instead of full 'Products' title.", "has_issue": true, "region": { "height": 68, "width": 69, "x": 191, "y": 125 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Main heading clipped on left showing 'ducts' and 'IANAGEMENT' instead of full 'Products' title.", "has_issue": true, "region": { "height": 68, "width": 69, "x": 191, "y": 125 } } | 10.11 s |
| commerce-catalog-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.76 s |
| commerce-catalog-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 3.51 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 2 | réussite | Cible atteinte | 4.69 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 2 | réussite | Cible atteinte | 3.86 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 2 | réussite | Cible atteinte | 4.04 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 2 | réussite | Cible atteinte | 7.95 s |
| analytics-dashboard-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } | 4.03 s |
| analytics-dashboard-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card heading truncated at container edge, reading 'chann' instead of full text.", "has_issue": true, "region": { "height": 30, "width": 328, "x": 200, "y": 249 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card heading truncated at container edge, reading 'chann' instead of full text.", "has_issue": true, "region": { "height": 30, "width": 328, "x": 200, "y": 249 } } | 9.23 s |
| analytics-dashboard-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.09 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.88 s |
| iot-control-desktop/clean/grounding/devices-nav | 2 | réussite | Cible atteinte | 4.10 s |
| iot-control-desktop/clean/grounding/restart-device | 2 | réussite | Cible atteinte | 3.19 s |
| iot-control-desktop/clean/grounding/auto-mode | 2 | réussite | Cible atteinte | 10.00 s |
| iot-control-desktop/clean/grounding/pump-card | 2 | réussite | Cible atteinte | 2.78 s |
| iot-control-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect detected.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect detected.", "has_issue": false, "region": null } | 5.68 s |
| iot-control-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No visible layout defects detected.", "has_issue": false, "region": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No visible layout defects detected.", "has_issue": false, "region": null } | 8.11 s |
| iot-control-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.31 s |
| iot-control-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.58 s |
| banking-ios/clean/grounding/profile | 2 | réussite | Cible atteinte | 4.57 s |
| banking-ios/clean/grounding/transaction-search | 2 | réussite | Cible atteinte | 4.25 s |
| banking-ios/clean/grounding/hide-balance | 2 | réussite | Cible atteinte | 8.61 s |
| banking-ios/clean/grounding/transfer | 2 | réussite | Cible atteinte | 3.44 s |
| banking-ios/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed", "has_issue": false, "region": null } | 3.40 s |
| banking-ios/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "The 'Banking' header text is cropped at the left viewport edge, cutting off the first letter.", "has_issue": true, "region": { "height": 55, "width": 250, "x": 30, "y": 80 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "The 'Banking' header text is cropped at the left viewport edge, cutting off the first letter.", "has_issue": true, "region": { "height": 55, "width": 250, "x": 30, "y": 80 } } | 5.50 s |
| banking-ios/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 8.35 s |
| banking-ios/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.99 s |
| flight-pass-ios/clean/grounding/boarding-pass | 2 | réussite | Cible manquée | 4.49 s |
| flight-pass-ios/clean/grounding/check-in | 2 | réussite | Cible atteinte | 4.15 s |
| flight-pass-ios/clean/grounding/trips-tab | 2 | réussite | Cible atteinte | 6.30 s |
| flight-pass-ios/clean/grounding/profile-tab | 2 | réussite | Cible atteinte | 5.09 s |
| flight-pass-ios/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } | 4.56 s |
| flight-pass-ios/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red Flight update banner overlaps blue banner behind it, obscuring its text.", "has_issue": true, "region": { "height": 55, "width": 910, "x": 45, "y": 875 } } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red Flight update banner overlaps blue banner behind it, obscuring its text.", "has_issue": true, "region": { "height": 55, "width": 910, "x": 45, "y": 875 } } | 10.19 s |
| flight-pass-ios/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.61 s |
| flight-pass-ios/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.10 s |
| smart-home-android/clean/grounding/more | 2 | réussite | Cible atteinte | 6.58 s |
| smart-home-android/clean/grounding/room-search | 2 | réussite | Cible atteinte | 4.87 s |
| smart-home-android/clean/grounding/thermostat-card | 2 | réussite | Cible atteinte | 4.45 s |
| smart-home-android/clean/grounding/add-device | 2 | réussite | Cible atteinte | 4.54 s |
| smart-home-android/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect.", "has_issue": false, "region": null } | 4.85 s |
| smart-home-android/defect/layout-defect | 2 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": null, "description": "No layout defect detected", "has_issue": false, "region": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": null, "description": "No layout defect detected", "has_issue": false, "region": null } | 9.62 s |
| smart-home-android/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 3.87 s |
| smart-home-android/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.64 s |
| commerce-catalog-desktop/clean/grounding/search | 3 | réussite | Cible atteinte | 4.92 s |
| commerce-catalog-desktop/clean/grounding/add-product | 3 | réussite | Cible atteinte | 3.59 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 3 | réussite | Cible atteinte | 3.02 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 3 | réussite | Cible atteinte | 5.15 s |
| commerce-catalog-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect.", "has_issue": false, "region": null } | 4.52 s |
| commerce-catalog-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Page heading clipped on left edge showing 'ducts' instead of 'Products' with kicker text cut off.", "has_issue": true, "region": { "height": 68, "width": 69, "x": 190, "y": 124 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Page heading clipped on left edge showing 'ducts' instead of 'Products' with kicker text cut off.", "has_issue": true, "region": { "height": 68, "width": 69, "x": 190, "y": 124 } } | 5.27 s |
| commerce-catalog-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.90 s |
| commerce-catalog-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 4.55 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 3 | réussite | Cible atteinte | 5.64 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 3 | réussite | Cible atteinte | 3.06 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 3 | réussite | Cible atteinte | 3.71 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 3 | réussite | Cible atteinte | 5.52 s |
| analytics-dashboard-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect detected.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect detected.", "has_issue": false, "region": null } | 3.62 s |
| analytics-dashboard-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card heading truncated to 'Monthly revenue across all sales chann', clipped at container edge.", "has_issue": true, "region": { "height": 30, "width": 330, "x": 200, "y": 250 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card heading truncated to 'Monthly revenue across all sales chann', clipped at container edge.", "has_issue": true, "region": { "height": 30, "width": 330, "x": 200, "y": 250 } } | 8.87 s |
| analytics-dashboard-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.07 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.90 s |
| iot-control-desktop/clean/grounding/devices-nav | 3 | réussite | Cible atteinte | 3.49 s |
| iot-control-desktop/clean/grounding/restart-device | 3 | réussite | Cible atteinte | 4.07 s |
| iot-control-desktop/clean/grounding/auto-mode | 3 | réussite | Cible atteinte | 5.22 s |
| iot-control-desktop/clean/grounding/pump-card | 3 | réussite | Cible atteinte | 5.83 s |
| iot-control-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect detected.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect detected.", "has_issue": false, "region": null } | 4.75 s |
| iot-control-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No layout defect detected", "has_issue": false, "region": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No layout defect detected", "has_issue": false, "region": null } | 7.46 s |
| iot-control-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 4.12 s |
| iot-control-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.87 s |
| banking-ios/clean/grounding/profile | 3 | réussite | Cible atteinte | 4.00 s |
| banking-ios/clean/grounding/transaction-search | 3 | réussite | Cible atteinte | 5.64 s |
| banking-ios/clean/grounding/hide-balance | 3 | réussite | Cible atteinte | 6.21 s |
| banking-ios/clean/grounding/transfer | 3 | réussite | Cible atteinte | 3.53 s |
| banking-ios/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } | 5.26 s |
| banking-ios/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "Header title Banking and top brand text truncated at left viewport edge with B cropped.", "has_issue": true, "region": { "height": 75, "width": 270, "x": 10, "y": 35 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "Header title Banking and top brand text truncated at left viewport edge with B cropped.", "has_issue": true, "region": { "height": 75, "width": 270, "x": 10, "y": 35 } } | 7.26 s |
| banking-ios/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.34 s |
| banking-ios/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 3.49 s |
| flight-pass-ios/clean/grounding/boarding-pass | 3 | réussite | Cible manquée | 3.76 s |
| flight-pass-ios/clean/grounding/check-in | 3 | réussite | Cible atteinte | 4.72 s |
| flight-pass-ios/clean/grounding/trips-tab | 3 | réussite | Cible atteinte | 4.97 s |
| flight-pass-ios/clean/grounding/profile-tab | 3 | réussite | Cible atteinte | 5.65 s |
| flight-pass-ios/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect detected.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect detected.", "has_issue": false, "region": null } | 5.01 s |
| flight-pass-ios/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red flight update banner overlaps blue banner behind it, obscuring its text", "has_issue": true, "region": { "height": 75, "width": 920, "x": 40, "y": 840 } } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red flight update banner overlaps blue banner behind it, obscuring its text", "has_issue": true, "region": { "height": 75, "width": 920, "x": 40, "y": 840 } } | 5.54 s |
| flight-pass-ios/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.05 s |
| flight-pass-ios/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.99 s |
| smart-home-android/clean/grounding/more | 3 | réussite | Cible atteinte | 10.73 s |
| smart-home-android/clean/grounding/room-search | 3 | réussite | Cible atteinte | 4.38 s |
| smart-home-android/clean/grounding/thermostat-card | 3 | réussite | Cible atteinte | 4.20 s |
| smart-home-android/clean/grounding/add-device | 3 | réussite | Cible atteinte | 3.82 s |
| smart-home-android/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defects observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defects observed.", "has_issue": false, "region": null } | 4.26 s |
| smart-home-android/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": "broken_responsive_reflow", "description": "The Kitchen lights card is indented and narrower than other cards, breaking list alignment.", "has_issue": true, "region": { "height": 90, "width": 675, "x": 280, "y": 530 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": "broken_responsive_reflow", "description": "The Kitchen lights card is indented and narrower than other cards, breaking list alignment.", "has_issue": true, "region": { "height": 90, "width": 675, "x": 280, "y": 530 } } | 11.76 s |
| smart-home-android/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.94 s |
| smart-home-android/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.33 s |
GPT-5.6 Luna · 144 exécutions de cas
Versions de modèle renvoyées : openai/gpt-5.6-luna.
Cibles avec le même résultat atteint/manqué sur les trois répétitions : 23 / 24. La concordance peut aussi signifier qu’une cible est manquée à chaque fois.
| Cas | Répétition | Résultat | Issue | Temps |
|---|---|---|---|---|
| commerce-catalog-desktop/clean/grounding/search | 1 | réussite | Cible atteinte | 2.56 s |
| commerce-catalog-desktop/clean/grounding/add-product | 1 | réussite | Cible atteinte | 2.00 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 1 | réussite | Cible atteinte | 2.45 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 1 | réussite | Cible atteinte | 2.89 s |
| commerce-catalog-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.03 s |
| commerce-catalog-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 127, "width": 70, "height": 62 }, "description": "The left portions of the page heading text are visibly clipped, showing “IAGEMENT” and “ducts” instead of the full labels." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 127, "width": 70, "height": 62 }, "description": "The left portions of the page heading text are visibly clipped, showing “IAGEMENT” and “ducts” instead of the full labels." } | 4.66 s |
| commerce-catalog-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.85 s |
| commerce-catalog-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.73 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 1 | réussite | Cible atteinte | 5.28 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 1 | réussite | Cible atteinte | 2.12 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 1 | réussite | Cible atteinte | 1.87 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 1 | réussite | Cible atteinte | 3.44 s |
| analytics-dashboard-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.97 s |
| analytics-dashboard-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 250, "width": 326, "height": 34 }, "description": "The first summary card's heading is visibly truncated at the card's right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 250, "width": 326, "height": 34 }, "description": "The first summary card's heading is visibly truncated at the card's right edge." } | 3.08 s |
| analytics-dashboard-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.31 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.29 s |
| iot-control-desktop/clean/grounding/devices-nav | 1 | réussite | Cible atteinte | 2.47 s |
| iot-control-desktop/clean/grounding/restart-device | 1 | réussite | Cible atteinte | 2.03 s |
| iot-control-desktop/clean/grounding/auto-mode | 1 | réussite | Cible atteinte | 4.02 s |
| iot-control-desktop/clean/grounding/pump-card | 1 | réussite | Cible atteinte | 2.64 s |
| iot-control-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.91 s |
| iot-control-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 3.79 s |
| iot-control-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.55 s |
| iot-control-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.29 s |
| banking-ios/clean/grounding/profile | 1 | réussite | Cible atteinte | 2.82 s |
| banking-ios/clean/grounding/transaction-search | 1 | réussite | Cible manquée | 1.87 s |
| banking-ios/clean/grounding/hide-balance | 1 | réussite | Cible atteinte | 3.78 s |
| banking-ios/clean/grounding/transfer | 1 | réussite | Cible atteinte | 1.69 s |
| banking-ios/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.95 s |
| banking-ios/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 44, "y": 71, "width": 244, "height": 33 }, "description": "The left edge of the large “Banking” heading is visibly cropped, leaving the initial character partially cut off." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 44, "y": 71, "width": 244, "height": 33 }, "description": "The left edge of the large “Banking” heading is visibly cropped, leaving the initial character partially cut off." } | 6.58 s |
| banking-ios/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.64 s |
| banking-ios/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.33 s |
| flight-pass-ios/clean/grounding/boarding-pass | 1 | réussite | Cible atteinte | 2.10 s |
| flight-pass-ios/clean/grounding/check-in | 1 | réussite | Cible atteinte | 2.62 s |
| flight-pass-ios/clean/grounding/trips-tab | 1 | réussite | Cible atteinte | 4.46 s |
| flight-pass-ios/clean/grounding/profile-tab | 1 | réussite | Cible atteinte | 5.20 s |
| flight-pass-ios/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.61 s |
| flight-pass-ios/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "A red flight update banner visibly covers the lower portion of a blue actionable button near the bottom of the screen." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "A red flight update banner visibly covers the lower portion of a blue actionable button near the bottom of the screen." } | 3.27 s |
| flight-pass-ios/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.40 s |
| flight-pass-ios/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.14 s |
| smart-home-android/clean/grounding/more | 1 | réussite | Cible atteinte | 2.51 s |
| smart-home-android/clean/grounding/room-search | 1 | réussite | Cible atteinte | 1.95 s |
| smart-home-android/clean/grounding/thermostat-card | 1 | réussite | Cible atteinte | 2.74 s |
| smart-home-android/clean/grounding/add-device | 1 | réussite | Cible atteinte | 2.44 s |
| smart-home-android/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.88 s |
| smart-home-android/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 510, "width": 600, "height": 69 }, "description": "The second Quick view card is visibly offset to the right and does not align with the other cards in the list." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 510, "width": 600, "height": 69 }, "description": "The second Quick view card is visibly offset to the right and does not align with the other cards in the list." } | 5.45 s |
| smart-home-android/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.57 s |
| smart-home-android/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.57 s |
| commerce-catalog-desktop/clean/grounding/search | 2 | réussite | Cible atteinte | 2.62 s |
| commerce-catalog-desktop/clean/grounding/add-product | 2 | réussite | Cible atteinte | 2.64 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 2 | réussite | Cible atteinte | 2.37 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 2 | réussite | Cible atteinte | 3.21 s |
| commerce-catalog-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.91 s |
| commerce-catalog-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 192, "y": 151, "width": 72, "height": 39 }, "description": "The page title is visibly clipped on the left, displaying only “ducts” instead of the full title." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 192, "y": 151, "width": 72, "height": 39 }, "description": "The page title is visibly clipped on the left, displaying only “ducts” instead of the full title." } | 4.33 s |
| commerce-catalog-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.17 s |
| commerce-catalog-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.70 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 2 | réussite | Cible atteinte | 4.50 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 2 | réussite | Cible atteinte | 2.67 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 2 | réussite | Cible atteinte | 2.36 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 2 | réussite | Cible atteinte | 3.04 s |
| analytics-dashboard-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.84 s |
| analytics-dashboard-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 252, "width": 325, "height": 28 }, "description": "The first summary card's headline is visibly truncated at the right edge, ending mid-word." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 252, "width": 325, "height": 28 }, "description": "The first summary card's headline is visibly truncated at the right edge, ending mid-word." } | 2.88 s |
| analytics-dashboard-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.57 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.66 s |
| iot-control-desktop/clean/grounding/devices-nav | 2 | réussite | Cible atteinte | 3.05 s |
| iot-control-desktop/clean/grounding/restart-device | 2 | réussite | Cible atteinte | 2.25 s |
| iot-control-desktop/clean/grounding/auto-mode | 2 | réussite | Cible atteinte | 2.80 s |
| iot-control-desktop/clean/grounding/pump-card | 2 | réussite | Cible atteinte | 3.40 s |
| iot-control-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.31 s |
| iot-control-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.85 s |
| iot-control-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 2.08 s |
| iot-control-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 2.07 s |
| banking-ios/clean/grounding/profile | 2 | réussite | Cible atteinte | 3.15 s |
| banking-ios/clean/grounding/transaction-search | 2 | réussite | Cible atteinte | 2.17 s |
| banking-ios/clean/grounding/hide-balance | 2 | réussite | Cible atteinte | 3.60 s |
| banking-ios/clean/grounding/transfer | 2 | réussite | Cible atteinte | 2.21 s |
| banking-ios/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.69 s |
| banking-ios/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 45, "y": 70, "width": 245, "height": 32 }, "description": "The left edge of the “Banking” header text is visibly clipped, with the initial letter cut off." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 45, "y": 70, "width": 245, "height": 32 }, "description": "The left edge of the “Banking” header text is visibly clipped, with the initial letter cut off." } | 4.84 s |
| banking-ios/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.59 s |
| banking-ios/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.55 s |
| flight-pass-ios/clean/grounding/boarding-pass | 2 | réussite | Cible atteinte | 3.22 s |
| flight-pass-ios/clean/grounding/check-in | 2 | réussite | Cible atteinte | 2.06 s |
| flight-pass-ios/clean/grounding/trips-tab | 2 | réussite | Cible atteinte | 7.16 s |
| flight-pass-ios/clean/grounding/profile-tab | 2 | réussite | Cible atteinte | 3.82 s |
| flight-pass-ios/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.55 s |
| flight-pass-ios/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 64 }, "description": "A red flight-update banner visibly overlaps and obscures the blue actionable button near the bottom of the screen." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 64 }, "description": "A red flight-update banner visibly overlaps and obscures the blue actionable button near the bottom of the screen." } | 3.37 s |
| flight-pass-ios/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.73 s |
| flight-pass-ios/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.64 s |
| smart-home-android/clean/grounding/more | 2 | réussite | Cible atteinte | 2.67 s |
| smart-home-android/clean/grounding/room-search | 2 | réussite | Cible atteinte | 2.30 s |
| smart-home-android/clean/grounding/thermostat-card | 2 | réussite | Cible atteinte | 2.26 s |
| smart-home-android/clean/grounding/add-device | 2 | réussite | Cible atteinte | 1.93 s |
| smart-home-android/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.57 s |
| smart-home-android/defect/layout-defect | 2 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.87 s |
| smart-home-android/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.17 s |
| smart-home-android/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.36 s |
| commerce-catalog-desktop/clean/grounding/search | 3 | réussite | Cible atteinte | 2.61 s |
| commerce-catalog-desktop/clean/grounding/add-product | 3 | réussite | Cible atteinte | 2.88 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 3 | réussite | Cible atteinte | 4.31 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 3 | réussite | Cible atteinte | 3.36 s |
| commerce-catalog-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.36 s |
| commerce-catalog-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 76, "height": 64 }, "description": "The page title is visibly clipped on the left, displaying “ducts” instead of the full “Products”." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 76, "height": 64 }, "description": "The page title is visibly clipped on the left, displaying “ducts” instead of the full “Products”." } | 3.66 s |
| commerce-catalog-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.69 s |
| commerce-catalog-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.80 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 3 | réussite | Cible atteinte | 3.61 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 3 | réussite | Cible atteinte | 2.33 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 3 | réussite | Cible atteinte | 2.09 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 3 | réussite | Cible atteinte | 3.14 s |
| analytics-dashboard-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.33 s |
| analytics-dashboard-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 253, "width": 326, "height": 28 }, "description": "The first summary card's heading is visibly cut off at the right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 253, "width": 326, "height": 28 }, "description": "The first summary card's heading is visibly cut off at the right edge." } | 3.52 s |
| analytics-dashboard-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.61 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.64 s |
| iot-control-desktop/clean/grounding/devices-nav | 3 | réussite | Cible atteinte | 2.44 s |
| iot-control-desktop/clean/grounding/restart-device | 3 | réussite | Cible atteinte | 3.00 s |
| iot-control-desktop/clean/grounding/auto-mode | 3 | réussite | Cible atteinte | 3.80 s |
| iot-control-desktop/clean/grounding/pump-card | 3 | réussite | Cible atteinte | 3.33 s |
| iot-control-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.57 s |
| iot-control-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 463, "y": 558, "width": 222, "height": 66 }, "description": "A maintenance notification overlays the device table content." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 463, "y": 558, "width": 222, "height": 66 }, "description": "A maintenance notification overlays the device table content." } | 3.65 s |
| iot-control-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.82 s |
| iot-control-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.48 s |
| banking-ios/clean/grounding/profile | 3 | réussite | Cible atteinte | 2.94 s |
| banking-ios/clean/grounding/transaction-search | 3 | réussite | Cible atteinte | 2.50 s |
| banking-ios/clean/grounding/hide-balance | 3 | réussite | Cible atteinte | 4.97 s |
| banking-ios/clean/grounding/transfer | 3 | réussite | Cible atteinte | 2.55 s |
| banking-ios/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 2.96 s |
| banking-ios/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 44, "y": 72, "width": 244, "height": 29 }, "description": "The left edge of the Banking heading is visibly clipped, cutting off the first character." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 44, "y": 72, "width": 244, "height": 29 }, "description": "The left edge of the Banking heading is visibly clipped, cutting off the first character." } | 3.82 s |
| banking-ios/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.26 s |
| banking-ios/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.30 s |
| flight-pass-ios/clean/grounding/boarding-pass | 3 | réussite | Cible atteinte | 2.52 s |
| flight-pass-ios/clean/grounding/check-in | 3 | réussite | Cible atteinte | 2.24 s |
| flight-pass-ios/clean/grounding/trips-tab | 3 | réussite | Cible atteinte | 4.28 s |
| flight-pass-ios/clean/grounding/profile-tab | 3 | réussite | Cible atteinte | 3.22 s |
| flight-pass-ios/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.70 s |
| flight-pass-ios/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 63 }, "description": "A red flight update banner obscures the lower portion of a blue actionable button near the bottom of the screen." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 63 }, "description": "A red flight update banner obscures the lower portion of a blue actionable button near the bottom of the screen." } | 3.14 s |
| flight-pass-ios/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.40 s |
| flight-pass-ios/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.32 s |
| smart-home-android/clean/grounding/more | 3 | réussite | Cible atteinte | 2.35 s |
| smart-home-android/clean/grounding/room-search | 3 | réussite | Cible atteinte | 2.65 s |
| smart-home-android/clean/grounding/thermostat-card | 3 | réussite | Cible atteinte | 2.64 s |
| smart-home-android/clean/grounding/add-device | 3 | réussite | Cible atteinte | 1.92 s |
| smart-home-android/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 1.88 s |
| smart-home-android/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 510, "width": 600, "height": 69 }, "description": "The second Quick view card is misaligned and compressed to the right compared with the other full-width cards." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 510, "width": 600, "height": 69 }, "description": "The second Quick view card is misaligned and compressed to the right compared with the other full-width cards." } | 4.82 s |
| smart-home-android/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.19 s |
| smart-home-android/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 1.62 s |
Jev 1.14 · 144 exécutions de cas
Versions de modèle renvoyées : typesafe/jev-1.14-20261014.
Cibles avec le même résultat atteint/manqué sur les trois répétitions : 24 / 24. La concordance peut aussi signifier qu’une cible est manquée à chaque fois.
| Cas | Répétition | Résultat | Issue | Temps |
|---|---|---|---|---|
| commerce-catalog-desktop/clean/grounding/search | 1 | réussite | Cible atteinte | 0.43 s |
| commerce-catalog-desktop/clean/grounding/add-product | 1 | réussite | Cible atteinte | 0.41 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 1 | réussite | Cible atteinte | 0.40 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 1 | réussite | Cible atteinte | 0.38 s |
| commerce-catalog-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.47 s |
| commerce-catalog-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 190, "y": 140, "width": 60, "height": 69.99999999999999 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 190, "y": 140, "width": 60, "height": 69.99999999999999 }, "description": "" } | 0.39 s |
| commerce-catalog-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.35 s |
| commerce-catalog-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.44 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 1 | réussite | Cible atteinte | 0.53 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 1 | réussite | Cible atteinte | 0.37 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 1 | réussite | Cible atteinte | 0.38 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 1 | réussite | Cible atteinte | 0.34 s |
| analytics-dashboard-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.48 s |
| analytics-dashboard-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 360.00000000000006, "height": 110.00000000000001 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 360.00000000000006, "height": 110.00000000000001 }, "description": "" } | 0.36 s |
| analytics-dashboard-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.33 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.39 s |
| iot-control-desktop/clean/grounding/devices-nav | 1 | réussite | Cible atteinte | 0.34 s |
| iot-control-desktop/clean/grounding/restart-device | 1 | réussite | Cible atteinte | 0.32 s |
| iot-control-desktop/clean/grounding/auto-mode | 1 | réussite | Cible atteinte | 0.41 s |
| iot-control-desktop/clean/grounding/pump-card | 1 | réussite | Cible atteinte | 0.37 s |
| iot-control-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.48 s |
| iot-control-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 450, "y": 550, "width": 239.99999999999994, "height": 89.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 450, "y": 550, "width": 239.99999999999994, "height": 89.99999999999997 }, "description": "" } | 0.38 s |
| iot-control-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.35 s |
| iot-control-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.46 s |
| banking-ios/clean/grounding/profile | 1 | réussite | Cible atteinte | 0.39 s |
| banking-ios/clean/grounding/transaction-search | 1 | réussite | Cible atteinte | 0.48 s |
| banking-ios/clean/grounding/hide-balance | 1 | réussite | Cible atteinte | 0.41 s |
| banking-ios/clean/grounding/transfer | 1 | réussite | Cible atteinte | 0.43 s |
| banking-ios/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.37 s |
| banking-ios/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" } | 0.30 s |
| banking-ios/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.37 s |
| banking-ios/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.38 s |
| flight-pass-ios/clean/grounding/boarding-pass | 1 | réussite | Cible atteinte | 0.31 s |
| flight-pass-ios/clean/grounding/check-in | 1 | réussite | Cible atteinte | 0.48 s |
| flight-pass-ios/clean/grounding/trips-tab | 1 | réussite | Cible atteinte | 0.34 s |
| flight-pass-ios/clean/grounding/profile-tab | 1 | réussite | Cible atteinte | 0.32 s |
| flight-pass-ios/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.30 s |
| flight-pass-ios/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 70.00000000000006 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 70.00000000000006 }, "description": "" } | 0.37 s |
| flight-pass-ios/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.28 s |
| flight-pass-ios/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.38 s |
| smart-home-android/clean/grounding/more | 1 | réussite | Cible atteinte | 0.36 s |
| smart-home-android/clean/grounding/room-search | 1 | réussite | Cible atteinte | 0.34 s |
| smart-home-android/clean/grounding/thermostat-card | 1 | réussite | Cible atteinte | 0.37 s |
| smart-home-android/clean/grounding/add-device | 1 | réussite | Cible atteinte | 0.33 s |
| smart-home-android/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.43 s |
| smart-home-android/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 470, "width": 590, "height": 99.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 470, "width": 590, "height": 99.99999999999997 }, "description": "" } | 0.39 s |
| smart-home-android/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.38 s |
| smart-home-android/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.41 s |
| commerce-catalog-desktop/clean/grounding/search | 2 | réussite | Cible atteinte | 0.44 s |
| commerce-catalog-desktop/clean/grounding/add-product | 2 | réussite | Cible atteinte | 0.42 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 2 | réussite | Cible atteinte | 0.38 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 2 | réussite | Cible atteinte | 0.42 s |
| commerce-catalog-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.38 s |
| commerce-catalog-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 180, "y": 140, "width": 70, "height": 69.99999999999999 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 180, "y": 140, "width": 70, "height": 69.99999999999999 }, "description": "" } | 0.39 s |
| commerce-catalog-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.38 s |
| commerce-catalog-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.54 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 2 | réussite | Cible atteinte | 0.41 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 2 | réussite | Cible atteinte | 0.66 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 2 | réussite | Cible atteinte | 0.48 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 2 | réussite | Cible atteinte | 0.37 s |
| analytics-dashboard-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.39 s |
| analytics-dashboard-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 360.00000000000006, "height": 100 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 360.00000000000006, "height": 100 }, "description": "" } | 0.39 s |
| analytics-dashboard-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.36 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.87 s |
| iot-control-desktop/clean/grounding/devices-nav | 2 | réussite | Cible atteinte | 0.46 s |
| iot-control-desktop/clean/grounding/restart-device | 2 | réussite | Cible atteinte | 0.66 s |
| iot-control-desktop/clean/grounding/auto-mode | 2 | réussite | Cible atteinte | 0.39 s |
| iot-control-desktop/clean/grounding/pump-card | 2 | réussite | Cible atteinte | 0.35 s |
| iot-control-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.33 s |
| iot-control-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 460, "y": 540, "width": 239.99999999999994, "height": 99.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 460, "y": 540, "width": 239.99999999999994, "height": 99.99999999999997 }, "description": "" } | 0.37 s |
| iot-control-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.33 s |
| iot-control-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.38 s |
| banking-ios/clean/grounding/profile | 2 | réussite | Cible atteinte | 0.33 s |
| banking-ios/clean/grounding/transaction-search | 2 | réussite | Cible atteinte | 0.30 s |
| banking-ios/clean/grounding/hide-balance | 2 | réussite | Cible atteinte | 0.33 s |
| banking-ios/clean/grounding/transfer | 2 | réussite | Cible atteinte | 0.41 s |
| banking-ios/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.30 s |
| banking-ios/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" } | 0.44 s |
| banking-ios/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.31 s |
| banking-ios/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.29 s |
| flight-pass-ios/clean/grounding/boarding-pass | 2 | réussite | Cible atteinte | 0.30 s |
| flight-pass-ios/clean/grounding/check-in | 2 | réussite | Cible atteinte | 0.29 s |
| flight-pass-ios/clean/grounding/trips-tab | 2 | réussite | Cible atteinte | 0.27 s |
| flight-pass-ios/clean/grounding/profile-tab | 2 | réussite | Cible atteinte | 0.31 s |
| flight-pass-ios/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.32 s |
| flight-pass-ios/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 70.00000000000006 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 70.00000000000006 }, "description": "" } | 0.31 s |
| flight-pass-ios/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.29 s |
| flight-pass-ios/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.29 s |
| smart-home-android/clean/grounding/more | 2 | réussite | Cible atteinte | 0.32 s |
| smart-home-android/clean/grounding/room-search | 2 | réussite | Cible atteinte | 0.30 s |
| smart-home-android/clean/grounding/thermostat-card | 2 | réussite | Cible atteinte | 0.38 s |
| smart-home-android/clean/grounding/add-device | 2 | réussite | Cible atteinte | 0.32 s |
| smart-home-android/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.29 s |
| smart-home-android/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 480, "width": 590, "height": 99.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 480, "width": 590, "height": 99.99999999999997 }, "description": "" } | 0.30 s |
| smart-home-android/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.32 s |
| smart-home-android/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.35 s |
| commerce-catalog-desktop/clean/grounding/search | 3 | réussite | Cible atteinte | 0.35 s |
| commerce-catalog-desktop/clean/grounding/add-product | 3 | réussite | Cible atteinte | 0.37 s |
| commerce-catalog-desktop/clean/grounding/catalog-query | 3 | réussite | Cible atteinte | 0.40 s |
| commerce-catalog-desktop/clean/grounding/stock-filter | 3 | réussite | Cible atteinte | 0.45 s |
| commerce-catalog-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.37 s |
| commerce-catalog-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 180, "y": 140, "width": 70, "height": 69.99999999999999 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 180, "y": 140, "width": 70, "height": 69.99999999999999 }, "description": "" } | 0.38 s |
| commerce-catalog-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.39 s |
| commerce-catalog-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.43 s |
| analytics-dashboard-desktop/clean/grounding/audience-tab | 3 | réussite | Cible atteinte | 0.33 s |
| analytics-dashboard-desktop/clean/grounding/revenue-card | 3 | réussite | Cible atteinte | 0.35 s |
| analytics-dashboard-desktop/clean/grounding/search-field | 3 | réussite | Cible atteinte | 0.36 s |
| analytics-dashboard-desktop/clean/grounding/compare-toggle | 3 | réussite | Cible atteinte | 0.38 s |
| analytics-dashboard-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.35 s |
| analytics-dashboard-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 350.00000000000006, "height": 110.00000000000001 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 350.00000000000006, "height": 110.00000000000001 }, "description": "" } | 0.36 s |
| analytics-dashboard-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.36 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.39 s |
| iot-control-desktop/clean/grounding/devices-nav | 3 | réussite | Cible atteinte | 0.33 s |
| iot-control-desktop/clean/grounding/restart-device | 3 | réussite | Cible atteinte | 0.32 s |
| iot-control-desktop/clean/grounding/auto-mode | 3 | réussite | Cible atteinte | 0.34 s |
| iot-control-desktop/clean/grounding/pump-card | 3 | réussite | Cible atteinte | 0.35 s |
| iot-control-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.37 s |
| iot-control-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 460, "y": 550, "width": 229.99999999999991, "height": 89.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 460, "y": 550, "width": 229.99999999999991, "height": 89.99999999999997 }, "description": "" } | 0.37 s |
| iot-control-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.36 s |
| iot-control-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.44 s |
| banking-ios/clean/grounding/profile | 3 | réussite | Cible atteinte | 0.29 s |
| banking-ios/clean/grounding/transaction-search | 3 | réussite | Cible atteinte | 0.29 s |
| banking-ios/clean/grounding/hide-balance | 3 | réussite | Cible atteinte | 0.29 s |
| banking-ios/clean/grounding/transfer | 3 | réussite | Cible atteinte | 0.33 s |
| banking-ios/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.44 s |
| banking-ios/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" } | 0.30 s |
| banking-ios/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.28 s |
| banking-ios/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.44 s |
| flight-pass-ios/clean/grounding/boarding-pass | 3 | réussite | Cible atteinte | 0.32 s |
| flight-pass-ios/clean/grounding/check-in | 3 | réussite | Cible atteinte | 0.30 s |
| flight-pass-ios/clean/grounding/trips-tab | 3 | réussite | Cible atteinte | 0.29 s |
| flight-pass-ios/clean/grounding/profile-tab | 3 | réussite | Cible atteinte | 0.28 s |
| flight-pass-ios/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.36 s |
| flight-pass-ios/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 60.00000000000006 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 60.00000000000006 }, "description": "" } | 0.30 s |
| flight-pass-ios/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.29 s |
| flight-pass-ios/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.29 s |
| smart-home-android/clean/grounding/more | 3 | réussite | Cible atteinte | 0.29 s |
| smart-home-android/clean/grounding/room-search | 3 | réussite | Cible atteinte | 0.39 s |
| smart-home-android/clean/grounding/thermostat-card | 3 | réussite | Cible atteinte | 0.35 s |
| smart-home-android/clean/grounding/add-device | 3 | réussite | Cible atteinte | 0.35 s |
| smart-home-android/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } | 0.35 s |
| smart-home-android/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 480, "width": 590, "height": 99.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 480, "width": 590, "height": 99.99999999999997 }, "description": "" } | 0.53 s |
| smart-home-android/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.39 s |
| smart-home-android/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.34 s |
GPT-6 Luna Decisions · 72 exécutions de cas
Versions de modèle renvoyées : openai/gpt-6-luna-decisions-20261006.
| Cas | Répétition | Résultat | Issue | Temps |
|---|---|---|---|---|
| commerce-catalog-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.38 s |
| commerce-catalog-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null } | 0.45 s |
| commerce-catalog-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.32 s |
| commerce-catalog-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.35 s |
| analytics-dashboard-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.38 s |
| analytics-dashboard-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" } | 0.36 s |
| analytics-dashboard-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.30 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.38 s |
| iot-control-desktop/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.32 s |
| iot-control-desktop/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" } | 0.52 s |
| iot-control-desktop/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.55 s |
| iot-control-desktop/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.26 s |
| banking-ios/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.24 s |
| banking-ios/defect/layout-defect | 1 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null } | 0.21 s |
| banking-ios/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.29 s |
| banking-ios/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.22 s |
| flight-pass-ios/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.53 s |
| flight-pass-ios/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" } | 0.66 s |
| flight-pass-ios/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.38 s |
| flight-pass-ios/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.19 s |
| smart-home-android/clean/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.33 s |
| smart-home-android/defect/layout-defect | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" } | 0.27 s |
| smart-home-android/clean/element-presence/present | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.26 s |
| smart-home-android/clean/element-presence/absent | 1 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.21 s |
| commerce-catalog-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.35 s |
| commerce-catalog-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null } | 0.38 s |
| commerce-catalog-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.27 s |
| commerce-catalog-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.36 s |
| analytics-dashboard-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.35 s |
| analytics-dashboard-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" } | 0.32 s |
| analytics-dashboard-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.47 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.28 s |
| iot-control-desktop/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.74 s |
| iot-control-desktop/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" } | 0.60 s |
| iot-control-desktop/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.91 s |
| iot-control-desktop/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.41 s |
| banking-ios/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.24 s |
| banking-ios/defect/layout-defect | 2 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null } | 0.21 s |
| banking-ios/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.49 s |
| banking-ios/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.24 s |
| flight-pass-ios/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.62 s |
| flight-pass-ios/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" } | 0.27 s |
| flight-pass-ios/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.26 s |
| flight-pass-ios/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.23 s |
| smart-home-android/clean/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.24 s |
| smart-home-android/defect/layout-defect | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" } | 0.53 s |
| smart-home-android/clean/element-presence/present | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.29 s |
| smart-home-android/clean/element-presence/absent | 2 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.19 s |
| commerce-catalog-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.49 s |
| commerce-catalog-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null } | 0.25 s |
| commerce-catalog-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.50 s |
| commerce-catalog-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.48 s |
| analytics-dashboard-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.36 s |
| analytics-dashboard-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" } | 0.23 s |
| analytics-dashboard-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 1.32 s |
| analytics-dashboard-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.52 s |
| iot-control-desktop/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.36 s |
| iot-control-desktop/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" } | 0.37 s |
| iot-control-desktop/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.31 s |
| iot-control-desktop/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.28 s |
| banking-ios/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.25 s |
| banking-ios/defect/layout-defect | 3 | réussite | The model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null } | 0.25 s |
| banking-ios/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.27 s |
| banking-ios/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.52 s |
| flight-pass-ios/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 0.54 s |
| flight-pass-ios/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" } | 1.22 s |
| flight-pass-ios/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.19 s |
| flight-pass-ios/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.89 s |
| smart-home-android/clean/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } | 1.09 s |
| smart-home-android/defect/layout-defect | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" } | 0.25 s |
| smart-home-android/clean/element-presence/present | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true } | 0.21 s |
| smart-home-android/clean/element-presence/absent | 3 | réussite | The model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false } | 0.32 s |
Le pilote terminé
Exécutions de cas enregistrées : 1368 / 1368. Tentatives HTTP : 1368. Coût API déclaré : $3.3802.
Neuf configurations peuvent fournir des coordonnées cibles. GPT-6 Luna Decisions répond aux questions de classification et est exclu des résultats de localisation.
Méthodologie
Mesurer ce que le modèle voit réellement
Les sections consacrées aux expériences documentent leurs propres règles d’évaluation. Cette méthodologie commune couvre le contrat d’entrée partagé, les opérations des fournisseurs, le contexte de recherche et les limites.
EntréeCapture d’écran uniquement
Géométrie0–1000
ExécutionsTrois par cas
ÉvaluationMétriques distinctes
Opérations
Temps de réponse, échecs et coût
Aller-retour HTTP et analyse locale des tentatives finales retenues, sans interrogation des métadonnées de génération. Toutes les tentatives de transport sont incluses dans le coût déclaré. La configuration de classification seule exécute deux fois moins de cas.
| Modèle | Terminés / prévus | Échecs finaux des requêtes | Réponses invalides | Médiane | P95 | Coût déclaré | Coût pour 1 000 requêtes |
|---|---|---|---|---|---|---|---|
| GPT-6 Luna | 144 / 144 | 0 | 0 | 2.18 s | 3.83 s | $0.0187 | $0.1296 |
| GPT-6.1 Sol | 144 / 144 | 0 | 0 | 2.68 s | 4.88 s | $0.2803 | $1.9468 |
| Claude Sonnet 5.5 | 144 / 144 | 0 | 8 | 2.20 s | 3.99 s | $0.5803 | $4.0297 |
| Gemini 3.8 Flash | 144 / 144 | 0 | 1 | 2.42 s | 10.96 s | $0.2655 | $1.8437 |
| Grok 4.7 | 144 / 144 | 0 | 2 | 12.03 s | 28.38 s | $1.1171 | $7.7580 |
| Qwen3.8 Max (0902) | 144 / 144 | 0 | 1 | 8.14 s | 14.76 s | $0.6158 | $4.2762 |
| Muse Spark 1.3 | 144 / 144 | 0 | 0 | 4.22 s | 9.18 s | $0.4342 | $3.0151 |
| GPT-5.6 Luna | 144 / 144 | 0 | 0 | 2.34 s | 4.80 s | $0.0440 | $0.3057 |
| Jev 1.14 | 144 / 144 | 0 | 0 | 0.37 s | 0.48 s | $0.0159 | $0.1104 |
| GPT-6 Luna Decisions | 72 / 72 | 0 | 0 | 0.34 s | 0.90 s | $0.0085 | $0.1177 |
Vérifications de présence : la même question oui/non
Comparez le temps de réponse et le coût sur les mêmes cas de présence. Les requêtes de mise en page des modèles généraux exigent aussi une région ; leur charge de travail diffère donc de celle de Decisions limité à la classification.
| Modèle | Cas de présence | Médiane | P95 | Coût de présence pour 1 000 requêtes |
|---|---|---|---|---|
| GPT-6 Luna | 36 | 1.45 s | 1.88 s | $0.0752 |
| GPT-6.1 Sol | 36 | 2.18 s | 2.68 s | $1.4487 |
| Claude Sonnet 5.5 | 36 | 1.85 s | 2.78 s | $2.9955 |
| Gemini 3.8 Flash | 36 | 2.45 s | 2.98 s | $1.2008 |
| Grok 4.7 | 36 | 1.97 s | 5.95 s | $2.8797 |
| Qwen3.8 Max (0902) | 36 | 4.51 s | 6.79 s | $2.6708 |
| Muse Spark 1.3 | 36 | 2.06 s | 4.23 s | $2.0156 |
| GPT-5.6 Luna | 36 | 1.58 s | 2.11 s | $0.1533 |
| Jev 1.14 | 36 | 0.37 s | 0.48 s | $0.1069 |
| GPT-6 Luna Decisions | 36 | 0.31 s | 0.89 s | $0.1062 |
Méthodologie
Ce qui a changé et ce qui est resté figé
- Les six captures, annotations, consignes et schémas de réponse d’origine proviennent du corpus du 13/08/2026. La nouvelle édition en conserve ses propres copies.
- Chaque cas est exécuté trois fois. Les modèles génératifs demandent un effort de raisonnement faible et une limite de sortie de 2 048 tokens, contre 400 auparavant. Les fournisseurs peuvent déclarer des tokens de raisonnement facturés au-delà de cette limite. Cette modification des conditions d’exécution est explicitement signalée.
- Les requêtes sont fixées sur des points de terminaison standard opérationnels des fournisseurs d’origine, sans routage de secours. Les points de terminaison, prix, paramètres et empreintes exacts sont conservés.
- Les modèles décisionnels reçoivent des questions natives en plus de la tâche. Les probabilités Noul utilisent un seuil fixe de 0,5. Le point de clic de Jev est calculé au centre du rectangle ; les autres modèles capables de renvoyer une géométrie prédisent un point de clic distinct.
- Luna Decisions prend en charge la classification, mais pas les sorties Box. Son évaluation de mise en page mesure donc uniquement la présence et la catégorie des défauts. Une sortie native typée et la validité du JSON créé par un adaptateur ne sont pas équivalentes à la génération libre de JSON.
- Les réponses invalides ne sont ni corrigées ni relancées. Seuls les échecs de transport temporaires peuvent faire l’objet d’une nouvelle tentative ; chaque tentative est conservée. Les variations du fournisseur et du réseau peuvent affecter les temps de réponse. Les coûts déclarés incluent les remises automatiques de cache ; les répétitions suivantes peuvent réutiliser des caches chauds.
Trois répétitions mesurent la cohérence sur ces cas. Six écrans synthétiques ne permettent pas d’établir une supériorité générale, une fiabilité en production ou une capacité de navigation en plusieurs étapes. Aucune action sur appareil, récupération ou réalisation d’objectif de navigation n’est testée.
Contrat d’entrée
Capture d’écran uniquement
Texte de prompt et un PNG en base64 uniquement — sans DOM, arbre d’accessibilité, code source, OCR, outils, plugins ni recherche Web.
Rendu fixe
Chromium en 1440 × 900 ou 390 × 844 ; facteur d’échelle de l’appareil : 1.
Géométrie normalisée
Vérité terrain dérivée du navigateur dans un espace de coordonnées de 0 à 1000.
Paires stables
Les variantes saines et défectueuses préservent la géométrie cible et les hachages des captures.
Opérations
Garder le comportement du fournisseur visible
Identifiants de modèle fixés, routage vers les fournisseurs d’origine, replis désactivés et contrats de réponse propres à chaque tâche explicités.
Politique de nouvelle tentative
Une nouvelle tentative pour une erreur 429, 5xx ou réseau — pas pour les mauvaises réponses.
Métriques opérationnelles
La latence, les échecs, les nouvelles tentatives et le coût restent distincts de l’exactitude.
Confidentialité
Les enregistrements publiés omettent les clés et les identifiants de requête des fournisseurs.
Données et licence
Télécharger cette édition
Prévalidation du modèle ↓Plan d’exécution ↓Manifeste des captures ↓Résultats et scores individuels ↓Tentatives brutes conservées ↓Constats et inventaire des échecs ↓
Les captures et résultats agrégés créés par Repeato sont disponibles sous CC BY 4.0. Les sorties brutes des modèles restent soumises aux conditions des fournisseurs. Consultez la notice de licence.