DescargarReservar una demostración

Investigación de Repeato · Benchmark de visión de LLM

¿Se puede confiar en los LLM multimodales para las pruebas de interfaz de usuario?

Benchmark basado únicamente en capturas de pantalla para la automatización de pruebas de interfaz: localizar elementos accionables, reconocer cuándo falta un elemento, detectar defectos visibles de diseño y registrar el coste operativo de obtener una respuesta.

2026-10-17

Seis pantallas deterministas. Veinticuatro objetivos de localización. 10 configuraciones de API. Tres repeticiones por caso. Un piloto acotado, no una clasificación.

Investigación de Stephan PetzlPublicado

Pantallas6
Objetivos de comprensión24
Categorías de defectos6
Configuraciones de API10

Resultados del piloto

Qué muestra esta edición

Tres repeticiones por caso en seis pantallas sintéticas revelan diferencias que merecen más investigación. No establecen clasificaciones generales ni fiabilidad en producción.

01

Localizar controles con precisión

GPT-6.1 Sol acertó los 72 objetivos en tres repeticiones, con un solapamiento medio del 81,2 % del área objetivo. Cuatro configuraciones acertaron todos los objetivos; la precisión de sus áreas predichas varió.

03

Detectar un defecto es solo el primer paso

Jev identificó correctamente las 36 pantallas correctas o defectuosas, pero nombró el defecto exacto en 12 de 18 intentos con pantallas defectuosas. Estos intentos repiten tres veces seis defectos distintos.

Qué significa esto para la automatización de pruebas

Usa el tipo de visión adecuado para cada paso de la prueba

La interacción exacta, la observación continua y la interpretación semántica tienen requisitos de rendimiento diferentes. No se debe esperar que una sola técnica de visión optimice las tres.

Observación localHasta 20 veces por segundoSolicitud a un modelo remotoMediana de 0,3 a 12 segundos en este pilotoCómo combinamos la visión local y los LLMContraer análisis

El bucle local de visión de Repeato puede inspeccionar la pantalla a 20 Hz, cada 50 ms. Este piloto midió tiempos de respuesta remotos medianos de unos 0,3 a 12 segundos según la tarea. La tarea compartida de presencia permite una comparación más justa; una clasificación rápida no implica una localización igual de rápida. Estas cifras locales y remotas proceden de mediciones distintas, no de una comparación directa controlada.

01

Feedback rápido para interacciones repetidas

Las huellas visuales locales pueden buscar continuamente los objetivos grabados y actuar en cuanto la pantalla esté lista. Repetir llamadas a modelos remotos con la misma frecuencia no sería práctico.

02

Predecible, privado y económico

La coincidencia local no requiere un recorrido de ida y vuelta por la red, disponibilidad del proveedor ni cargos de API por solicitud. Las pantallas permanecen en la estación de trabajo y el mismo algoritmo puede ejecutarse sin conexión.

03

IA para las preguntas que la coincidencia no puede responder

La visión basada en LLM justifica su latencia cuando una prueba necesita criterio semántico: interpretar texto o números, comprender contenido dinámico o comprobar conceptos como una gráfica con tendencia ascendente.

División práctica

Usa la visión por computador local para el bucle de control de alta frecuencia: encontrar objetivos, esperar, interactuar y comparar visualmente. Recurre selectivamente a la visión mediante IA cuando la aserción requiera realmente comprensión lingüística o semántica.

Experimentos

Tres tareas distintas de visión de la interfaz de usuario

La localización, la presencia de elementos y la integridad de la interfaz responden a preguntas diferentes y utilizan métricas independientes. El piloto no las combina en una puntuación general.

Experimento 1 · Localización de la interfaz de usuario

¿Puede un modelo localizar el elemento de interfaz solicitado?

24 objetivos × tres repeticiones = 72 intentos por configuración con capacidad geométrica. Las respuestas inválidas y fallidas cuentan como fallos.

Nueve configuraciones pueden devolver coordenadas de objetivos. GPT-6 Luna Decisions responde a preguntas de clasificación y se excluye de los resultados de localización.

IoU de las cajasMás alto es mejor · escala de 0 a 1
  1. GPT-6 Luna0.796
  2. GPT-6.1 Sol0.812
  3. Claude Sonnet 5.50.748
  4. Gemini 3.8 Flash0.733
  5. Grok 4.70.477
  6. Qwen3.8 Max (0902)0.760
  7. Muse Spark 1.30.630
  8. GPT-5.6 Luna0.786
  9. Jev 1.140.581
Coste totalMenor es mejor · 72 solicitudes
  1. GPT-6 Luna$0.0096
  2. GPT-6.1 Sol$0.1447
  3. Claude Sonnet 5.5$0.2970
  4. Gemini 3.8 Flash$0.1007
  5. Grok 4.7$0.6647
  6. Qwen3.8 Max (0902)$0.3254
  7. Muse Spark 1.3$0.2220
  8. GPT-5.6 Luna$0.0258
  9. Jev 1.14$0.0079
Latencia mediaMás bajo es mejor · por solicitud
  1. GPT-6 Luna2.35 s
  2. GPT-6.1 Sol2.91 s
  3. Claude Sonnet 5.52.32 s
  4. Gemini 3.8 Flash2.75 s
  5. Grok 4.716.91 s
  6. Qwen3.8 Max (0902)8.89 s
  7. Muse Spark 1.34.80 s
  8. GPT-5.6 Luna2.93 s
  9. Jev 1.140.37 s
Comparación entre precisión y costeMás arriba y más a la izquierda es mejor.
Better value0.000.250.500.751.00$0.00$0.14$0.28$0.42$0.56$0.70Total cost for 72 requests (USD) →Box IoU →GPT-6 LunaGPT-6.1 SolClaude Sonnet 5.5Gemini 3.8 FlashGrok 4.7Qwen3.8 Max (0902)Muse Spark 1.3GPT-5.6 LunaJev 1.14
GPT-6 Lunaopenai/gpt-6-luna →
Aprobadas99% de aciertos0.7962.35 s$0.0096
GPT-6.1 Solopenai/gpt-6.1-sol →
Aprobadas100 % de aciertos0.8122.91 s$0.1447
Claude Sonnet 5.5anthropic/claude-sonnet-5.5 →
Respuesta no válida97% de aciertos0.7482.32 s$0.2970
Gemini 3.8 Flashgoogle/gemini-3.8-flash →
Aprobadas100 % de aciertos0.7332.75 s$0.1007
Grok 4.7x-ai/grok-4.7 →
Respuesta no válida72% de aciertos0.47716.91 s$0.6647
Qwen3.8 Max (0902)qwen/qwen3.8-max-0902 →
Aprobadas100 % de aciertos0.7608.89 s$0.3254
Muse Spark 1.3meta/muse-spark-1.3 →
Aprobadas94% de aciertos0.6304.80 s$0.2220
GPT-5.6 Lunaopenai/gpt-5.6-luna →
Aprobadas99% de aciertos0.7862.93 s$0.0258
Jev 1.14typesafe/jev-1.14 →
Aprobadas100 % de aciertos0.5810.37 s$0.0079

Metodología

Cómo se puntúa la localización de la interfaz de usuario

Cuatro objetivos por captura de pantalla, correspondientes a tipos comunes de elementos de UI accionables.

Precisión de puntos

El punto de clic predicho queda dentro del cuadro objetivo.

Precisión de cajas

IoU medio y proporción de predicciones con IoU ≥ 0.5.

Validez

Tasa de respuestas válidas según el esquema; tres repeticiones por caso revelan la consistencia de cada caso.

Experimento 2 · Presencia de elementos

¿Puede un modelo determinar cuándo un elemento no está presente?

Seis objetivos presentes y seis ausentes, comprobados tres veces cada uno. Se evalúa la presencia visible, no la consecución de un objetivo de navegación.

GPT-6 Lunaopenai/gpt-6-luna →
100 % válidos36/36 casos · 0 intentos fallidos100%100%100%0%1.50 s$0.0752por cada 1.000 casos conservados
GPT-6.1 Solopenai/gpt-6.1-sol →
100 % válidos36/36 casos · 0 intentos fallidos100%100%100%0%2.24 s$1.4487por cada 1.000 casos conservados
Claude Sonnet 5.5anthropic/claude-sonnet-5.5 →
100 % válidos36/36 casos · 0 intentos fallidos100%100%100%0%2.02 s$2.9955por cada 1.000 casos conservados
Gemini 3.8 Flashgoogle/gemini-3.8-flash →
100 % válidos36/36 casos · 0 intentos fallidos100%100%100%0%2.44 s$1.2008por cada 1.000 casos conservados
Grok 4.7x-ai/grok-4.7 →
100 % válidos36/36 casos · 0 intentos fallidos100%100%100%0%2.66 s$2.8797por cada 1.000 casos conservados
Qwen3.8 Max (0902)qwen/qwen3.8-max-0902 →
100 % válidos36/36 casos · 0 intentos fallidos100%100%100%0%4.81 s$2.6708por cada 1.000 casos conservados
Muse Spark 1.3meta/muse-spark-1.3 →
100 % válidos36/36 casos · 0 intentos fallidos100%100%100%0%2.47 s$2.0156por cada 1.000 casos conservados
GPT-5.6 Lunaopenai/gpt-5.6-luna →
100 % válidos36/36 casos · 0 intentos fallidos100%100%100%0%1.59 s$0.1533por cada 1.000 casos conservados
Jev 1.14typesafe/jev-1.14 →
100 % válidos36/36 casos · 0 intentos fallidos100%100%100%0%0.38 s$0.1069por cada 1.000 casos conservados
GPT-6 Luna Decisionsopenai/gpt-6-luna-decisions →
100 % válidos36/36 casos · 0 intentos fallidos100%100%100%0%0.39 s$0.1062por cada 1.000 casos conservados

Tres repeticiones por caso en seis pantallas sintéticas. Los costes incluyen los descuentos de caché declarados. Estos resultados no establecen fiabilidad en producción ni clasificaciones generales de modelos.

Metodología

Cómo se puntúa la presencia de elementos

Una consulta cuya presencia se conoce y otra plausible pero ausente por cada pantalla limpia.

Detección

La exactitud equilibrada asigna el mismo peso a los casos presentes y ausentes.

Comportamiento de la clase

La recuperación, la especificidad y la tasa de falsos positivos revelan la dirección del error.

Solo booleano

found=true o found=false; sin puntuación de localización.

Experimento 3 · Integridad de la interfaz de usuario

¿Puede un modelo distinguir entre diseños correctos y dañados?

Seis pantallas correctas y seis con defectos introducidos × tres repeticiones. Reconocer una pantalla defectuosa, identificar el defecto exacto y localizar su región son resultados separados.

Comparación entre precisión de detección de defectos y costeMás arriba y más a la izquierda es mejor.
Better value0.000.250.500.751.00$0.00$2.00$4.00$6.00$8.00$10.00Cost per 1,000 retained defect cases (USD) →Balanced accuracy →GPT-6 LunaGPT-6.1 SolClaude Sonnet 5.5Gemini 3.8 FlashGrok 4.7Qwen3.8 Max (0902)Muse Spark 1.3GPT-5.6 LunaJev 1.14GPT-6 Luna Decisions
Comparación entre precisión de detección de defectos y latenciaMás arriba y más a la izquierda es mejor. La latencia es el tiempo medio de respuesta por caso de defecto conservado. La variante Decisions de solo clasificación no produce regiones.
Better value0.000.250.500.751.000.0 s3.6 s7.2 s10.8 s14.4 s18.0 sMean latency (seconds) →Balanced accuracy →GPT-6 LunaGPT-6.1 SolClaude Sonnet 5.5Gemini 3.8 FlashGrok 4.7Qwen3.8 Max (0902)Muse Spark 1.3GPT-5.6 LunaJev 1.14GPT-6 Luna Decisions
GPT-6 Lunaopenai/gpt-6-luna →
100 % válidos36/36 casos · 0 intentos fallidos92%83%100%56%3.04 s$0.1751por cada 1.000 casos conservados
GPT-6.1 Solopenai/gpt-6.1-sol →
100 % válidos36/36 casos · 0 intentos fallidos92%83%100%33%3.64 s$2.3176por cada 1.000 casos conservados
Claude Sonnet 5.5anthropic/claude-sonnet-5.5 →
83 % válidos36/36 casos · 0 intentos fallidos83%67%100%50%2.87 s$4.8735por cada 1.000 casos conservados
Gemini 3.8 Flashgoogle/gemini-3.8-flash →
97% válidas36/36 casos · 0 intentos fallidos78%100%56%72%7.06 s$3.3765por cada 1.000 casos conservados
Grok 4.7x-ai/grok-4.7 →
97% válidas36/36 casos · 0 intentos fallidos94%94%94%72%17.48 s$9.6872por cada 1.000 casos conservados
Qwen3.8 Max (0902)qwen/qwen3.8-max-0902 →
97% válidas36/36 casos · 0 intentos fallidos94%94%94%39%11.78 s$5.3957por cada 1.000 casos conservados
Muse Spark 1.3meta/muse-spark-1.3 →
100 % válidos36/36 casos · 0 intentos fallidos89%78%100%56%6.34 s$3.8791por cada 1.000 casos conservados
GPT-5.6 Lunaopenai/gpt-5.6-luna →
100 % válidos36/36 casos · 0 intentos fallidos92%83%100%67%2.88 s$0.3518por cada 1.000 casos conservados
Jev 1.14typesafe/jev-1.14 →
100 % válidos36/36 casos · 0 intentos fallidos100%100%100%67%0.37 s$0.1166por cada 1.000 casos conservados
GPT-6 Luna Decisionsopenai/gpt-6-luna-decisions →
100 % válidos36/36 casos · 0 intentos fallidos83%67%100%50%0.42 s$0.1291por cada 1.000 casos conservados

Tres repeticiones por caso en seis pantallas sintéticas. Los costes incluyen los descuentos de caché declarados. Estos resultados no establecen fiabilidad en producción ni clasificaciones generales de modelos.

El evaluador existente de defectos asigna un solapamiento nulo cuando la categoría es incorrecta. Luna Decisions se evalúa solo en clasificación y no tiene que producir una región.

Metodología

Cómo se puntúa la integridad de la interfaz de usuario

Las escenas limpias y defectuosas emparejadas cubren seis tipos objetivos de fallos de diseño.

Detección

Exactitud equilibrada, recuperación, especificidad y tasa de falsos positivos.

Clasificación

Exactitud por categoría de defecto tras una detección correcta.

Localización

IoU del cuadro predicho frente a las regiones de defectos derivadas del navegador.

Corpus congelado

Tres pantallas de escritorio y tres móviles

Cada fixture tiene un control limpio y una variante deliberadamente defectuosa. Las capturas usan ventanas gráficas fijas, recursos locales, geometría estable de objetivos y límites exactos de diferencia de píxeles.

Control limpio del catálogo comercial
Catálogo comercial con recorte del contenedor
Catálogo de comercioWeb de escritorio · Recorte del contenedor
Control limpio del panel de analítica
Panel de analítica con desbordamiento de texto
Panel de analíticaWeb de escritorio · Desbordamiento de texto
Control limpio del centro de control de IoT
Centro de control de IoT con superposición de elementos
Centro de control de IoTWeb de escritorio · Superposición de elementos
Banca móvil con control limpio
Banca móvil con recorte de la ventana gráfica
Banca móvilSimilar a iOS · Recorte de la ventana gráfica
Pase de vuelo con control limpio
Pase de vuelo con acción oculta
Pase de vueloSimilar a iOS · Acción oculta
Casa inteligente con control limpio
Casa inteligente con reflujo adaptable defectuoso
Casa inteligenteSimilar a Android · Reflujo adaptable defectuoso

Expedientes de modelos

Examinar cada caso conservado

Todas las repeticiones siguen visibles, incluidos los intentos incorrectos, inválidos y fallidos. Despliega un modelo para examinar sus casos; la descarga de datos conserva las predicciones y decisiones nativas.

openaiGPT-6 LunaResultados completos del benchmark →openaiGPT-6.1 SolResultados completos del benchmark →anthropicClaude Sonnet 5.5Resultados completos del benchmark →googleGemini 3.8 FlashResultados completos del benchmark →x-aiGrok 4.7Resultados completos del benchmark →qwenQwen3.8 Max (0902)Resultados completos del benchmark →metaMuse Spark 1.3Resultados completos del benchmark →openaiGPT-5.6 LunaResultados completos del benchmark →typesafeJev 1.14Resultados completos del benchmark →openaiGPT-6 Luna DecisionsResultados completos del benchmark →
GPT-6 Luna · 144 ejecuciones de casos

Versiones de modelo devueltas: openai/gpt-6-luna.

Objetivos con el mismo resultado de acierto/fallo en las tres repeticiones: 23 / 24. La coincidencia también puede significar que un objetivo se falla siempre.

CasoRepeticiónResultadoDesenlaceTiempo
commerce-catalog-desktop/clean/grounding/search1éxitoObjetivo alcanzado3.33 s
commerce-catalog-desktop/clean/grounding/add-product1éxitoObjetivo alcanzado1.92 s
commerce-catalog-desktop/clean/grounding/catalog-query1éxitoObjetivo alcanzado2.65 s
commerce-catalog-desktop/clean/grounding/stock-filter1éxitoObjetivo alcanzado3.77 s
commerce-catalog-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.33 s
commerce-catalog-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 191, "y": 145, "width": 69, "height": 46 }, "description": "The page heading is clipped on the left, showing only “ducts” instead of the full title." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 191, "y": 145, "width": 69, "height": 46 }, "description": "The page heading is clipped on the left, showing only “ducts” instead of the full title." }3.98 s
commerce-catalog-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.55 s
commerce-catalog-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.28 s
analytics-dashboard-desktop/clean/grounding/audience-tab1éxitoObjetivo alcanzado2.66 s
analytics-dashboard-desktop/clean/grounding/revenue-card1éxitoObjetivo alcanzado2.42 s
analytics-dashboard-desktop/clean/grounding/search-field1éxitoObjetivo alcanzado2.01 s
analytics-dashboard-desktop/clean/grounding/compare-toggle1éxitoObjetivo alcanzado2.24 s
analytics-dashboard-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.99 s
analytics-dashboard-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 248, "width": 326, "height": 31 }, "description": "The revenue card heading is clipped at the card’s right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 248, "width": 326, "height": 31 }, "description": "The revenue card heading is clipped at the card’s right edge." }2.59 s
analytics-dashboard-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.57 s
analytics-dashboard-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.57 s
iot-control-desktop/clean/grounding/devices-nav1éxitoObjetivo alcanzado2.21 s
iot-control-desktop/clean/grounding/restart-device1éxitoObjetivo alcanzado1.98 s
iot-control-desktop/clean/grounding/auto-mode1éxitoObjetivo alcanzado2.62 s
iot-control-desktop/clean/grounding/pump-card1éxitoObjetivo alcanzado2.80 s
iot-control-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.04 s
iot-control-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 222, "height": 64 }, "description": "The maintenance notification overlaps the device table rows." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 222, "height": 64 }, "description": "The maintenance notification overlaps the device table rows." }3.62 s
iot-control-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.49 s
iot-control-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.57 s
banking-ios/clean/grounding/profile1éxitoObjetivo alcanzado2.46 s
banking-ios/clean/grounding/transaction-search1éxitoObjetivo alcanzado2.81 s
banking-ios/clean/grounding/hide-balance1éxitoObjetivo alcanzado2.59 s
banking-ios/clean/grounding/transfer1éxitoObjetivo alcanzado1.87 s
banking-ios/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.16 s
banking-ios/defect/layout-defect1éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }6.07 s
banking-ios/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.72 s
banking-ios/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.24 s
flight-pass-ios/clean/grounding/boarding-pass1éxitoObjetivo alcanzado2.49 s
flight-pass-ios/clean/grounding/check-in1éxitoObjetivo alcanzado2.78 s
flight-pass-ios/clean/grounding/trips-tab1éxitoObjetivo alcanzado1.63 s
flight-pass-ios/clean/grounding/profile-tab1éxitoObjetivo alcanzado2.53 s
flight-pass-ios/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.76 s
flight-pass-ios/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "A red flight-update banner covers the lower portion of the blue action button." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "A red flight-update banner covers the lower portion of the blue action button." }2.79 s
flight-pass-ios/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.24 s
flight-pass-ios/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.12 s
smart-home-android/clean/grounding/more1éxitoObjetivo alcanzado1.47 s
smart-home-android/clean/grounding/room-search1éxitoObjetivo alcanzado1.88 s
smart-home-android/clean/grounding/thermostat-card1éxitoObjetivo alcanzado1.46 s
smart-home-android/clean/grounding/add-device1éxitoObjetivo alcanzado1.38 s
smart-home-android/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.93 s
smart-home-android/defect/layout-defect1éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }4.04 s
smart-home-android/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.19 s
smart-home-android/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.21 s
commerce-catalog-desktop/clean/grounding/search2éxitoObjetivo alcanzado2.62 s
commerce-catalog-desktop/clean/grounding/add-product2éxitoObjetivo alcanzado2.12 s
commerce-catalog-desktop/clean/grounding/catalog-query2éxitoObjetivo alcanzado1.54 s
commerce-catalog-desktop/clean/grounding/stock-filter2éxitoObjetivo alcanzado2.71 s
commerce-catalog-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.88 s
commerce-catalog-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 151, "width": 68, "height": 44 }, "description": "The page heading is clipped on the left; only “ducts” is visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 151, "width": 68, "height": 44 }, "description": "The page heading is clipped on the left; only “ducts” is visible." }3.44 s
commerce-catalog-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.75 s
commerce-catalog-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.65 s
analytics-dashboard-desktop/clean/grounding/audience-tab2éxitoObjetivo alcanzado3.18 s
analytics-dashboard-desktop/clean/grounding/revenue-card2éxitoObjetivo alcanzado2.89 s
analytics-dashboard-desktop/clean/grounding/search-field2éxitoObjetivo alcanzado2.21 s
analytics-dashboard-desktop/clean/grounding/compare-toggle2éxitoObjetivo alcanzado2.67 s
analytics-dashboard-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.91 s
analytics-dashboard-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 253, "width": 326, "height": 37 }, "description": "The revenue card heading is cut off at the right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 253, "width": 326, "height": 37 }, "description": "The revenue card heading is cut off at the right edge." }2.65 s
analytics-dashboard-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.97 s
analytics-dashboard-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.67 s
iot-control-desktop/clean/grounding/devices-nav2éxitoObjetivo alcanzado2.20 s
iot-control-desktop/clean/grounding/restart-device2éxitoObjetivo alcanzado2.25 s
iot-control-desktop/clean/grounding/auto-mode2éxitoObjetivo alcanzado2.86 s
iot-control-desktop/clean/grounding/pump-card2éxitoObjetivo alcanzado3.25 s
iot-control-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.92 s
iot-control-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 222, "height": 64 }, "description": "The maintenance notification overlaps the device table rows." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 222, "height": 64 }, "description": "The maintenance notification overlaps the device table rows." }3.14 s
iot-control-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.77 s
iot-control-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.64 s
banking-ios/clean/grounding/profile2éxitoObjetivo alcanzado3.40 s
banking-ios/clean/grounding/transaction-search2éxitoObjetivo alcanzado2.17 s
banking-ios/clean/grounding/hide-balance2éxitoObjetivo alcanzado2.38 s
banking-ios/clean/grounding/transfer2éxitoObjetivo alcanzado1.75 s
banking-ios/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.87 s
banking-ios/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 72, "width": 240, "height": 32 }, "description": "The first letter of the Banking heading is clipped at the left edge of its container." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 72, "width": 240, "height": 32 }, "description": "The first letter of the Banking heading is clipped at the left edge of its container." }5.97 s
banking-ios/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.17 s
banking-ios/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.45 s
flight-pass-ios/clean/grounding/boarding-pass2éxitoObjetivo alcanzado2.26 s
flight-pass-ios/clean/grounding/check-in2éxitoObjetivo alcanzado1.47 s
flight-pass-ios/clean/grounding/trips-tab2éxitoObjetivo alcanzado1.62 s
flight-pass-ios/clean/grounding/profile-tab2éxitoObjetivo fallado1.55 s
flight-pass-ios/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.20 s
flight-pass-ios/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner obscures part of the blue actionable button beneath it." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner obscures part of the blue actionable button beneath it." }3.68 s
flight-pass-ios/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.22 s
flight-pass-ios/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.09 s
smart-home-android/clean/grounding/more2éxitoObjetivo alcanzado1.72 s
smart-home-android/clean/grounding/room-search2éxitoObjetivo alcanzado2.22 s
smart-home-android/clean/grounding/thermostat-card2éxitoObjetivo alcanzado1.53 s
smart-home-android/clean/grounding/add-device2éxitoObjetivo alcanzado1.48 s
smart-home-android/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.82 s
smart-home-android/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 354, "y": 510, "width": 646, "height": 69 }, "description": "The second Quick view card extends beyond the right edge of the viewport and is cropped." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 354, "y": 510, "width": 646, "height": 69 }, "description": "The second Quick view card extends beyond the right edge of the viewport and is cropped." }4.63 s
smart-home-android/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.18 s
smart-home-android/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.45 s
commerce-catalog-desktop/clean/grounding/search3éxitoObjetivo alcanzado2.81 s
commerce-catalog-desktop/clean/grounding/add-product3éxitoObjetivo alcanzado1.70 s
commerce-catalog-desktop/clean/grounding/catalog-query3éxitoObjetivo alcanzado2.45 s
commerce-catalog-desktop/clean/grounding/stock-filter3éxitoObjetivo alcanzado3.83 s
commerce-catalog-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }3.66 s
commerce-catalog-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 191, "y": 150, "width": 70, "height": 40 }, "description": "The page title is visibly clipped at its beginning, showing only “ducts”." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 191, "y": 150, "width": 70, "height": 40 }, "description": "The page title is visibly clipped at its beginning, showing only “ducts”." }3.67 s
commerce-catalog-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.85 s
commerce-catalog-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.33 s
analytics-dashboard-desktop/clean/grounding/audience-tab3éxitoObjetivo alcanzado3.45 s
analytics-dashboard-desktop/clean/grounding/revenue-card3éxitoObjetivo alcanzado2.06 s
analytics-dashboard-desktop/clean/grounding/search-field3éxitoObjetivo alcanzado2.00 s
analytics-dashboard-desktop/clean/grounding/compare-toggle3éxitoObjetivo alcanzado2.30 s
analytics-dashboard-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.75 s
analytics-dashboard-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 253, "width": 326, "height": 32 }, "description": "The monthly revenue card heading is clipped at the card’s right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 253, "width": 326, "height": 32 }, "description": "The monthly revenue card heading is clipped at the card’s right edge." }3.12 s
analytics-dashboard-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.27 s
analytics-dashboard-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.32 s
iot-control-desktop/clean/grounding/devices-nav3éxitoObjetivo alcanzado2.50 s
iot-control-desktop/clean/grounding/restart-device3éxitoObjetivo alcanzado2.75 s
iot-control-desktop/clean/grounding/auto-mode3éxitoObjetivo alcanzado2.79 s
iot-control-desktop/clean/grounding/pump-card3éxitoObjetivo alcanzado2.32 s
iot-control-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }3.03 s
iot-control-desktop/defect/layout-defect3éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }3.08 s
iot-control-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.44 s
iot-control-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.71 s
banking-ios/clean/grounding/profile3éxitoObjetivo alcanzado2.41 s
banking-ios/clean/grounding/transaction-search3éxitoObjetivo alcanzado2.90 s
banking-ios/clean/grounding/hide-balance3éxitoObjetivo alcanzado2.29 s
banking-ios/clean/grounding/transfer3éxitoObjetivo alcanzado2.51 s
banking-ios/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.21 s
banking-ios/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 0, "y": 69, "width": 285, "height": 35 }, "description": "The heading at the upper left is clipped on its left side." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 0, "y": 69, "width": 285, "height": 35 }, "description": "The heading at the upper left is clipped on its left side." }5.43 s
banking-ios/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.45 s
banking-ios/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.26 s
flight-pass-ios/clean/grounding/boarding-pass3éxitoObjetivo alcanzado2.16 s
flight-pass-ios/clean/grounding/check-in3éxitoObjetivo alcanzado2.23 s
flight-pass-ios/clean/grounding/trips-tab3éxitoObjetivo alcanzado2.66 s
flight-pass-ios/clean/grounding/profile-tab3éxitoObjetivo alcanzado3.25 s
flight-pass-ios/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.77 s
flight-pass-ios/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner obscures part of the blue actionable button beneath it." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner obscures part of the blue actionable button beneath it." }2.89 s
flight-pass-ios/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.24 s
flight-pass-ios/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.06 s
smart-home-android/clean/grounding/more3éxitoObjetivo alcanzado2.34 s
smart-home-android/clean/grounding/room-search3éxitoObjetivo alcanzado2.18 s
smart-home-android/clean/grounding/thermostat-card3éxitoObjetivo alcanzado1.72 s
smart-home-android/clean/grounding/add-device3éxitoObjetivo alcanzado1.51 s
smart-home-android/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.43 s
smart-home-android/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 354, "y": 510, "width": 646, "height": 69 }, "description": "The Kitchen lights card is cut off at the right edge of the viewport." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 354, "y": 510, "width": 646, "height": 69 }, "description": "The Kitchen lights card is cut off at the right edge of the viewport." }5.87 s
smart-home-android/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.47 s
smart-home-android/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.66 s
GPT-6.1 Sol · 144 ejecuciones de casos

Versiones de modelo devueltas: openai/gpt-6.1-sol.

Objetivos con el mismo resultado de acierto/fallo en las tres repeticiones: 24 / 24. La coincidencia también puede significar que un objetivo se falla siempre.

CasoRepeticiónResultadoDesenlaceTiempo
commerce-catalog-desktop/clean/grounding/search1éxitoObjetivo alcanzado3.73 s
commerce-catalog-desktop/clean/grounding/add-product1éxitoObjetivo alcanzado3.41 s
commerce-catalog-desktop/clean/grounding/catalog-query1éxitoObjetivo alcanzado2.75 s
commerce-catalog-desktop/clean/grounding/stock-filter1éxitoObjetivo alcanzado3.00 s
commerce-catalog-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }3.79 s
commerce-catalog-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 68, "height": 68 }, "description": "The page heading and its small label are clipped on the left, leaving only trailing portions of the text visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 68, "height": 68 }, "description": "The page heading and its small label are clipped on the left, leaving only trailing portions of the text visible." }5.65 s
commerce-catalog-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.04 s
commerce-catalog-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.52 s
analytics-dashboard-desktop/clean/grounding/audience-tab1éxitoObjetivo alcanzado4.02 s
analytics-dashboard-desktop/clean/grounding/revenue-card1éxitoObjetivo alcanzado3.04 s
analytics-dashboard-desktop/clean/grounding/search-field1éxitoObjetivo alcanzado3.42 s
analytics-dashboard-desktop/clean/grounding/compare-toggle1éxitoObjetivo alcanzado3.78 s
analytics-dashboard-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.68 s
analytics-dashboard-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 250, "width": 326, "height": 31 }, "description": "The monthly revenue card heading is clipped at the card's right edge." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 250, "width": 326, "height": 31 }, "description": "The monthly revenue card heading is clipped at the card's right edge." }3.75 s
analytics-dashboard-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.18 s
analytics-dashboard-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.13 s
iot-control-desktop/clean/grounding/devices-nav1éxitoObjetivo alcanzado3.30 s
iot-control-desktop/clean/grounding/restart-device1éxitoObjetivo alcanzado2.72 s
iot-control-desktop/clean/grounding/auto-mode1éxitoObjetivo alcanzado2.67 s
iot-control-desktop/clean/grounding/pump-card1éxitoObjetivo alcanzado2.61 s
iot-control-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.52 s
iot-control-desktop/defect/layout-defect1éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.67 s
iot-control-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.46 s
iot-control-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.27 s
banking-ios/clean/grounding/profile1éxitoObjetivo alcanzado2.88 s
banking-ios/clean/grounding/transaction-search1éxitoObjetivo alcanzado2.62 s
banking-ios/clean/grounding/hide-balance1éxitoObjetivo alcanzado2.78 s
banking-ios/clean/grounding/transfer1éxitoObjetivo alcanzado3.09 s
banking-ios/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.96 s
banking-ios/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 47, "width": 238, "height": 59 }, "description": "The brand label and Banking heading are clipped along their left edge, cutting off their initial letters." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 47, "width": 238, "height": 59 }, "description": "The brand label and Banking heading are clipped along their left edge, cutting off their initial letters." }8.64 s
banking-ios/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.87 s
banking-ios/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.35 s
flight-pass-ios/clean/grounding/boarding-pass1éxitoObjetivo alcanzado2.48 s
flight-pass-ios/clean/grounding/check-in1éxitoObjetivo alcanzado4.77 s
flight-pass-ios/clean/grounding/trips-tab1éxitoObjetivo alcanzado2.43 s
flight-pass-ios/clean/grounding/profile-tab1éxitoObjetivo alcanzado2.85 s
flight-pass-ios/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.39 s
flight-pass-ios/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 64 }, "description": "The flight update banner overlaps the bottom blue action button, obscuring its label and lower portion." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 64 }, "description": "The flight update banner overlaps the bottom blue action button, obscuring its label and lower portion." }3.86 s
flight-pass-ios/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.98 s
flight-pass-ios/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.53 s
smart-home-android/clean/grounding/more1éxitoObjetivo alcanzado2.27 s
smart-home-android/clean/grounding/room-search1éxitoObjetivo alcanzado2.44 s
smart-home-android/clean/grounding/thermostat-card1éxitoObjetivo alcanzado2.78 s
smart-home-android/clean/grounding/add-device1éxitoObjetivo alcanzado2.72 s
smart-home-android/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.44 s
smart-home-android/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 70 }, "description": "The Kitchen lights card is shifted right and clipped at the list's right edge, cutting off its right side and trailing control." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 70 }, "description": "The Kitchen lights card is shifted right and clipped at the list's right edge, cutting off its right side and trailing control." }5.47 s
smart-home-android/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.67 s
smart-home-android/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.36 s
commerce-catalog-desktop/clean/grounding/search2éxitoObjetivo alcanzado2.80 s
commerce-catalog-desktop/clean/grounding/add-product2éxitoObjetivo alcanzado2.61 s
commerce-catalog-desktop/clean/grounding/catalog-query2éxitoObjetivo alcanzado2.82 s
commerce-catalog-desktop/clean/grounding/stock-filter2éxitoObjetivo alcanzado3.15 s
commerce-catalog-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }3.00 s
commerce-catalog-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 73, "height": 68 }, "description": "The page heading and its section label are clipped on the left, hiding their initial characters." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 73, "height": 68 }, "description": "The page heading and its section label are clipped on the left, hiding their initial characters." }4.90 s
commerce-catalog-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.41 s
commerce-catalog-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.60 s
analytics-dashboard-desktop/clean/grounding/audience-tab2éxitoObjetivo alcanzado2.80 s
analytics-dashboard-desktop/clean/grounding/revenue-card2éxitoObjetivo alcanzado2.48 s
analytics-dashboard-desktop/clean/grounding/search-field2éxitoObjetivo alcanzado2.85 s
analytics-dashboard-desktop/clean/grounding/compare-toggle2éxitoObjetivo alcanzado3.92 s
analytics-dashboard-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.98 s
analytics-dashboard-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 249, "width": 326, "height": 32 }, "description": "The monthly revenue heading is cut off at the right edge of its card." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 249, "width": 326, "height": 32 }, "description": "The monthly revenue heading is cut off at the right edge of its card." }3.47 s
analytics-dashboard-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.60 s
analytics-dashboard-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.42 s
iot-control-desktop/clean/grounding/devices-nav2éxitoObjetivo alcanzado3.34 s
iot-control-desktop/clean/grounding/restart-device2éxitoObjetivo alcanzado2.94 s
iot-control-desktop/clean/grounding/auto-mode2éxitoObjetivo alcanzado4.14 s
iot-control-desktop/clean/grounding/pump-card2éxitoObjetivo alcanzado2.51 s
iot-control-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.47 s
iot-control-desktop/defect/layout-defect2éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.90 s
iot-control-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.94 s
iot-control-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.50 s
banking-ios/clean/grounding/profile2éxitoObjetivo alcanzado2.89 s
banking-ios/clean/grounding/transaction-search2éxitoObjetivo alcanzado2.58 s
banking-ios/clean/grounding/hide-balance2éxitoObjetivo alcanzado2.40 s
banking-ios/clean/grounding/transfer2éxitoObjetivo alcanzado2.81 s
banking-ios/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.89 s
banking-ios/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 69, "width": 241, "height": 38 }, "description": "The left edge of the Banking heading is clipped, cutting off part of its first letter." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 69, "width": 241, "height": 38 }, "description": "The left edge of the Banking heading is clipped, cutting off part of its first letter." }7.42 s
banking-ios/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.72 s
banking-ios/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.02 s
flight-pass-ios/clean/grounding/boarding-pass2éxitoObjetivo alcanzado3.14 s
flight-pass-ios/clean/grounding/check-in2éxitoObjetivo alcanzado2.86 s
flight-pass-ios/clean/grounding/trips-tab2éxitoObjetivo alcanzado2.24 s
flight-pass-ios/clean/grounding/profile-tab2éxitoObjetivo alcanzado2.47 s
flight-pass-ios/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.99 s
flight-pass-ios/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 64 }, "description": "The red flight-update banner overlaps the blue bottom action button, obscuring its label and much of its clickable area." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 64 }, "description": "The red flight-update banner overlaps the blue bottom action button, obscuring its label and much of its clickable area." }4.43 s
flight-pass-ios/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.94 s
flight-pass-ios/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.89 s
smart-home-android/clean/grounding/more2éxitoObjetivo alcanzado2.35 s
smart-home-android/clean/grounding/room-search2éxitoObjetivo alcanzado2.58 s
smart-home-android/clean/grounding/thermostat-card2éxitoObjetivo alcanzado2.82 s
smart-home-android/clean/grounding/add-device2éxitoObjetivo alcanzado2.71 s
smart-home-android/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.13 s
smart-home-android/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 71 }, "description": "The Kitchen lights row is shifted right and clipped at the content boundary, cutting off its right edge and trailing control." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 71 }, "description": "The Kitchen lights row is shifted right and clipped at the content boundary, cutting off its right edge and trailing control." }5.02 s
smart-home-android/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.05 s
smart-home-android/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.98 s
commerce-catalog-desktop/clean/grounding/search3éxitoObjetivo alcanzado3.93 s
commerce-catalog-desktop/clean/grounding/add-product3éxitoObjetivo alcanzado2.40 s
commerce-catalog-desktop/clean/grounding/catalog-query3éxitoObjetivo alcanzado3.39 s
commerce-catalog-desktop/clean/grounding/stock-filter3éxitoObjetivo alcanzado2.85 s
commerce-catalog-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the specified categories." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the specified categories." }3.40 s
commerce-catalog-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 70, "height": 68 }, "description": "The page heading and its overline are clipped on the left, leaving only trailing portions of the text visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 124, "width": 70, "height": 68 }, "description": "The page heading and its overline are clipped on the left, leaving only trailing portions of the text visible." }4.58 s
commerce-catalog-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.23 s
commerce-catalog-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }3.35 s
analytics-dashboard-desktop/clean/grounding/audience-tab3éxitoObjetivo alcanzado2.94 s
analytics-dashboard-desktop/clean/grounding/revenue-card3éxitoObjetivo alcanzado2.86 s
analytics-dashboard-desktop/clean/grounding/search-field3éxitoObjetivo alcanzado2.81 s
analytics-dashboard-desktop/clean/grounding/compare-toggle3éxitoObjetivo alcanzado3.48 s
analytics-dashboard-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.59 s
analytics-dashboard-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 249, "width": 327, "height": 32 }, "description": "The monthly revenue heading is cut off at the right edge of its card." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 201, "y": 249, "width": 327, "height": 32 }, "description": "The monthly revenue heading is cut off at the right edge of its card." }3.59 s
analytics-dashboard-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.55 s
analytics-dashboard-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.03 s
iot-control-desktop/clean/grounding/devices-nav3éxitoObjetivo alcanzado3.04 s
iot-control-desktop/clean/grounding/restart-device3éxitoObjetivo alcanzado3.26 s
iot-control-desktop/clean/grounding/auto-mode3éxitoObjetivo alcanzado2.99 s
iot-control-desktop/clean/grounding/pump-card3éxitoObjetivo alcanzado2.80 s
iot-control-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }3.12 s
iot-control-desktop/defect/layout-defect3éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.84 s
iot-control-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.39 s
iot-control-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.19 s
banking-ios/clean/grounding/profile3éxitoObjetivo alcanzado3.35 s
banking-ios/clean/grounding/transaction-search3éxitoObjetivo alcanzado2.37 s
banking-ios/clean/grounding/hide-balance3éxitoObjetivo alcanzado3.37 s
banking-ios/clean/grounding/transfer3éxitoObjetivo alcanzado2.54 s
banking-ios/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.05 s
banking-ios/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 44, "y": 68, "width": 245, "height": 39 }, "description": "The left edge of the Banking heading is clipped, cutting off part of its first letter." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 44, "y": 68, "width": 245, "height": 39 }, "description": "The left edge of the Banking heading is clipped, cutting off part of its first letter." }6.93 s
banking-ios/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.85 s
banking-ios/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.60 s
flight-pass-ios/clean/grounding/boarding-pass3éxitoObjetivo alcanzado2.12 s
flight-pass-ios/clean/grounding/check-in3éxitoObjetivo alcanzado2.21 s
flight-pass-ios/clean/grounding/trips-tab3éxitoObjetivo alcanzado3.53 s
flight-pass-ios/clean/grounding/profile-tab3éxitoObjetivo alcanzado2.35 s
flight-pass-ios/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.03 s
flight-pass-ios/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner covers the lower portion and label of the blue action button above the bottom navigation." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "The flight update banner covers the lower portion and label of the blue action button above the bottom navigation." }4.19 s
flight-pass-ios/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.91 s
flight-pass-ios/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.12 s
smart-home-android/clean/grounding/more3éxitoObjetivo alcanzado2.30 s
smart-home-android/clean/grounding/room-search3éxitoObjetivo alcanzado2.60 s
smart-home-android/clean/grounding/thermostat-card3éxitoObjetivo alcanzado2.75 s
smart-home-android/clean/grounding/add-device3éxitoObjetivo alcanzado2.49 s
smart-home-android/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.17 s
smart-home-android/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 70 }, "description": "The Kitchen lights card is shifted right and clipped at the list container’s right edge, cutting off its right side and trailing control." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 354, "y": 509, "width": 600, "height": 70 }, "description": "The Kitchen lights card is shifted right and clipped at the list container’s right edge, cutting off its right side and trailing control." }5.17 s
smart-home-android/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.06 s
smart-home-android/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.07 s
Claude Sonnet 5.5 · 144 ejecuciones de casos

Versiones de modelo devueltas: anthropic/claude-sonnet-5.5.

Objetivos con el mismo resultado de acierto/fallo en las tres repeticiones: 22 / 24. La coincidencia también puede significar que un objetivo se falla siempre.

CasoRepeticiónResultadoDesenlaceTiempo
commerce-catalog-desktop/clean/grounding/search1éxitoObjetivo alcanzado3.96 s
commerce-catalog-desktop/clean/grounding/add-product1respuesta inválidaObjetivo fallado2.02 s
commerce-catalog-desktop/clean/grounding/catalog-query1éxitoObjetivo alcanzado3.08 s
commerce-catalog-desktop/clean/grounding/stock-filter1éxitoObjetivo alcanzado2.40 s
commerce-catalog-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact and aligned." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact and aligned." }3.75 s
commerce-catalog-desktop/defect/layout-defect1respuesta inválidaThe answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 191, "y": 123, "width": 200, "height": 75 }, "description": "The page heading and eyebrow text are cut off on the left edge. The eyebrow reads 'IANAGEMENT' instead of 'MANAGEMENT', and the title reads 'ducts' instead of 'Products', so the leading characters are clipped by the container." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 191, "y": 123, "width": 200, "height": 75 }, "description": "The page heading and eyebrow text are cut off on the left edge. The eyebrow reads 'IANAGEMENT' instead of 'MANAGEMENT', and the title reads 'ducts' instead of 'Products', so the leading characters are clipped by the container." }4.65 s
commerce-catalog-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.46 s
commerce-catalog-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.24 s
analytics-dashboard-desktop/clean/grounding/audience-tab1éxitoObjetivo alcanzado2.47 s
analytics-dashboard-desktop/clean/grounding/revenue-card1éxitoObjetivo alcanzado2.46 s
analytics-dashboard-desktop/clean/grounding/search-field1éxitoObjetivo alcanzado2.94 s
analytics-dashboard-desktop/clean/grounding/compare-toggle1éxitoObjetivo alcanzado2.31 s
analytics-dashboard-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible clipping, overflow, overlap, obscured controls, or reflow problems; all text and controls are fully visible and properly contained." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible clipping, overflow, overlap, obscured controls, or reflow problems; all text and controls are fully visible and properly contained." }4.48 s
analytics-dashboard-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 250, "width": 356, "height": 32 }, "description": "The red card title 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is clipped by the container." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 250, "width": 356, "height": 32 }, "description": "The red card title 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is clipped by the container." }2.55 s
analytics-dashboard-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.14 s
analytics-dashboard-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.14 s
iot-control-desktop/clean/grounding/devices-nav1éxitoObjetivo alcanzado3.05 s
iot-control-desktop/clean/grounding/restart-device1éxitoObjetivo alcanzado2.13 s
iot-control-desktop/clean/grounding/auto-mode1éxitoObjetivo alcanzado2.39 s
iot-control-desktop/clean/grounding/pump-card1éxitoObjetivo alcanzado2.31 s
iot-control-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact." }3.98 s
iot-control-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' banner floats over the table rows, covering part of the Cooling pump and Intake valve row areas and overlapping the row divider." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' banner floats over the table rows, covering part of the Cooling pump and Intake valve row areas and overlapping the row divider." }4.73 s
iot-control-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.99 s
iot-control-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.35 s
banking-ios/clean/grounding/profile1éxitoObjetivo alcanzado1.84 s
banking-ios/clean/grounding/transaction-search1éxitoObjetivo alcanzado3.14 s
banking-ios/clean/grounding/hide-balance1éxitoObjetivo alcanzado1.92 s
banking-ios/clean/grounding/transfer1éxitoObjetivo alcanzado1.82 s
banking-ios/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all text and controls are fully visible, contained, and not overlapping." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all text and controls are fully visible, contained, and not overlapping." }2.21 s
banking-ios/defect/layout-defect1respuesta inválidaThe answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 300, "height": 115 }, "description": "The header title is cut off at the left screen edge: 'banking' renders as 'oanking' and the 'MAYA' label above it as 'IAYA', so the leading characters are cropped." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 300, "height": 115 }, "description": "The header title is cut off at the left screen edge: 'banking' renders as 'oanking' and the 'MAYA' label above it as 'IAYA', so the leading characters are cropped." }3.40 s
banking-ios/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.51 s
banking-ios/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.83 s
flight-pass-ios/clean/grounding/boarding-pass1éxitoObjetivo alcanzado2.46 s
flight-pass-ios/clean/grounding/check-in1éxitoObjetivo alcanzado1.91 s
flight-pass-ios/clean/grounding/trips-tab1éxitoObjetivo alcanzado1.89 s
flight-pass-ios/clean/grounding/profile-tab1éxitoObjetivo alcanzado2.30 s
flight-pass-ios/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; content is fully visible, text fits its containers, and no elements overlap or obscure controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; content is fully visible, text fits its containers, and no elements overlap or obscure controls." }2.34 s
flight-pass-ios/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 80 }, "description": "A red 'Flight update' banner covers the blue primary button near the bottom, hiding most of its label so only the tops of the letters remain visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 80 }, "description": "A red 'Flight update' banner covers the blue primary button near the bottom, hiding most of its label so only the tops of the letters remain visible." }3.21 s
flight-pass-ios/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.77 s
flight-pass-ios/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }4.33 s
smart-home-android/clean/grounding/more1éxitoObjetivo alcanzado2.23 s
smart-home-android/clean/grounding/room-search1éxitoObjetivo alcanzado9.81 s
smart-home-android/clean/grounding/thermostat-card1éxitoObjetivo alcanzado2.25 s
smart-home-android/clean/grounding/add-device1éxitoObjetivo alcanzado1.83 s
smart-home-android/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; all elements are fully rendered and aligned." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; all elements are fully rendered and aligned." }4.38 s
smart-home-android/defect/layout-defect1respuesta inválidaThe answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 533, "width": 646, "height": 46 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its right side is cut off." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 533, "width": 646, "height": 46 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its right side is cut off." }5.42 s
smart-home-android/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.53 s
smart-home-android/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.99 s
commerce-catalog-desktop/clean/grounding/search2éxitoObjetivo alcanzado2.13 s
commerce-catalog-desktop/clean/grounding/add-product2éxitoObjetivo alcanzado1.89 s
commerce-catalog-desktop/clean/grounding/catalog-query2éxitoObjetivo alcanzado2.72 s
commerce-catalog-desktop/clean/grounding/stock-filter2éxitoObjetivo alcanzado2.05 s
commerce-catalog-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." }2.41 s
commerce-catalog-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 125, "width": 205, "height": 57 }, "description": "The page heading and eyebrow text are cut off at the left edge of their container, showing 'IANAGEMENT' and 'ducts' instead of the full words." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 125, "width": 205, "height": 57 }, "description": "The page heading and eyebrow text are cut off at the left edge of their container, showing 'IANAGEMENT' and 'ducts' instead of the full words." }3.99 s
commerce-catalog-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.64 s
commerce-catalog-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.47 s
analytics-dashboard-desktop/clean/grounding/audience-tab2éxitoObjetivo alcanzado2.29 s
analytics-dashboard-desktop/clean/grounding/revenue-card2éxitoObjetivo alcanzado2.15 s
analytics-dashboard-desktop/clean/grounding/search-field2éxitoObjetivo alcanzado2.47 s
analytics-dashboard-desktop/clean/grounding/compare-toggle2éxitoObjetivo alcanzado2.02 s
analytics-dashboard-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible. The layout appears intact." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible. The layout appears intact." }3.55 s
analytics-dashboard-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 249, "width": 355, "height": 40 }, "description": "The red heading 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is clipped by the container." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 249, "width": 355, "height": 40 }, "description": "The red heading 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is clipped by the container." }2.22 s
analytics-dashboard-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.77 s
analytics-dashboard-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.68 s
iot-control-desktop/clean/grounding/devices-nav2éxitoObjetivo alcanzado2.64 s
iot-control-desktop/clean/grounding/restart-device2éxitoObjetivo alcanzado2.20 s
iot-control-desktop/clean/grounding/auto-mode2éxitoObjetivo alcanzado2.46 s
iot-control-desktop/clean/grounding/pump-card2éxitoObjetivo alcanzado2.21 s
iot-control-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." }2.09 s
iot-control-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "A red 'Maintenance scheduled' banner floats over the device table, covering parts of the Cooling pump and Intake valve rows and overlapping the row divider." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "A red 'Maintenance scheduled' banner floats over the device table, covering parts of the Cooling pump and Intake valve rows and overlapping the row divider." }3.48 s
iot-control-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.35 s
iot-control-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.89 s
banking-ios/clean/grounding/profile2éxitoObjetivo alcanzado2.23 s
banking-ios/clean/grounding/transaction-search2éxitoObjetivo alcanzado1.61 s
banking-ios/clean/grounding/hide-balance2éxitoObjetivo alcanzado1.87 s
banking-ios/clean/grounding/transfer2éxitoObjetivo alcanzado1.83 s
banking-ios/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all text, cards, and controls render fully within their containers without overlap or cropping." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all text, cards, and controls render fully within their containers without overlap or cropping." }2.29 s
banking-ios/defect/layout-defect2respuesta inválidaThe answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 300, "height": 55 }, "description": "The header text at the top left is cut off by the left edge of the viewport: the brand label reads 'IAYA' and the title reads 'anking' instead of 'Banking', so the first letters are cropped." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 300, "height": 55 }, "description": "The header text at the top left is cut off by the left edge of the viewport: the brand label reads 'IAYA' and the title reads 'anking' instead of 'Banking', so the first letters are cropped." }2.89 s
banking-ios/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.61 s
banking-ios/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.63 s
flight-pass-ios/clean/grounding/boarding-pass2éxitoObjetivo alcanzado2.21 s
flight-pass-ios/clean/grounding/check-in2éxitoObjetivo alcanzado2.25 s
flight-pass-ios/clean/grounding/trips-tab2éxitoObjetivo alcanzado1.87 s
flight-pass-ios/clean/grounding/profile-tab2éxitoObjetivo alcanzado2.24 s
flight-pass-ios/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; content is fully within the viewport, text fits its containers, and no elements overlap or obscure controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; content is fully within the viewport, text fits its containers, and no elements overlap or obscure controls." }2.20 s
flight-pass-ios/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 82 }, "description": "A red 'Flight update' banner overlaps the blue check-in button, hiding its label so the button text is cut off and the control is partially obscured." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 82 }, "description": "A red 'Flight update' banner overlaps the blue check-in button, hiding its label so the button text is cut off and the control is partially obscured." }2.08 s
flight-pass-ios/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.80 s
flight-pass-ios/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.71 s
smart-home-android/clean/grounding/more2éxitoObjetivo alcanzado1.91 s
smart-home-android/clean/grounding/room-search2éxitoObjetivo alcanzado1.75 s
smart-home-android/clean/grounding/thermostat-card2éxitoObjetivo alcanzado2.58 s
smart-home-android/clean/grounding/add-device2éxitoObjetivo alcanzado1.76 s
smart-home-android/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, elements are not overlapping, and nothing is cropped at the viewport edges." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, elements are not overlapping, and nothing is cropped at the viewport edges." }1.77 s
smart-home-android/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 531, "width": 628, "height": 55 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its chevron is cut off." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 531, "width": 628, "height": 55 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its chevron is cut off." }2.13 s
smart-home-android/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.74 s
smart-home-android/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.58 s
commerce-catalog-desktop/clean/grounding/search3éxitoObjetivo alcanzado2.58 s
commerce-catalog-desktop/clean/grounding/add-product3éxitoObjetivo alcanzado1.92 s
commerce-catalog-desktop/clean/grounding/catalog-query3éxitoObjetivo alcanzado3.00 s
commerce-catalog-desktop/clean/grounding/stock-filter3éxitoObjetivo alcanzado2.02 s
commerce-catalog-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact and aligned." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout is intact and aligned." }2.56 s
commerce-catalog-desktop/defect/layout-defect3respuesta inválidaThe answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 125, "width": 205, "height": 57 }, "description": "The page heading and eyebrow text are cut off at the left edge of their container. The eyebrow reads 'IANAGEMENT' instead of 'MANAGEMENT' and the title reads 'ducts' instead of 'Products'." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 125, "width": 205, "height": 57 }, "description": "The page heading and eyebrow text are cut off at the left edge of their container. The eyebrow reads 'IANAGEMENT' instead of 'MANAGEMENT' and the title reads 'ducts' instead of 'Products'." }3.32 s
commerce-catalog-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.68 s
commerce-catalog-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.86 s
analytics-dashboard-desktop/clean/grounding/audience-tab3éxitoObjetivo alcanzado2.65 s
analytics-dashboard-desktop/clean/grounding/revenue-card3éxitoObjetivo alcanzado1.85 s
analytics-dashboard-desktop/clean/grounding/search-field3éxitoObjetivo alcanzado2.33 s
analytics-dashboard-desktop/clean/grounding/compare-toggle3éxitoObjetivo alcanzado2.31 s
analytics-dashboard-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, nothing is cropped or overlapping, and controls are unobstructed." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, nothing is cropped or overlapping, and controls are unobstructed." }2.16 s
analytics-dashboard-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 199, "y": 243, "width": 347, "height": 50 }, "description": "The red heading 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is truncated by the container." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 199, "y": 243, "width": 347, "height": 50 }, "description": "The red heading 'Monthly revenue across all sales chann...' is cut off at the right edge of its card, so the text is truncated by the container." }2.14 s
analytics-dashboard-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.05 s
analytics-dashboard-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.21 s
iot-control-desktop/clean/grounding/devices-nav3éxitoObjetivo alcanzado2.43 s
iot-control-desktop/clean/grounding/restart-device3éxitoObjetivo alcanzado2.00 s
iot-control-desktop/clean/grounding/auto-mode3éxitoObjetivo alcanzado2.25 s
iot-control-desktop/clean/grounding/pump-card3éxitoObjetivo alcanzado1.91 s
iot-control-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, text overflow, overlap, obscured controls, or broken reflow is visible; the layout renders cleanly." }2.08 s
iot-control-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' banner floats over the table rows, covering part of the Cooling pump and Intake valve rows and breaking the table layout." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' banner floats over the table rows, covering part of the Cooling pump and Intake valve rows and breaking the table layout." }2.50 s
iot-control-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.88 s
iot-control-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.83 s
banking-ios/clean/grounding/profile3éxitoObjetivo alcanzado1.75 s
banking-ios/clean/grounding/transaction-search3éxitoObjetivo alcanzado2.71 s
banking-ios/clean/grounding/hide-balance3éxitoObjetivo alcanzado1.76 s
banking-ios/clean/grounding/transfer3éxitoObjetivo alcanzado1.72 s
banking-ios/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: text is fully contained, elements are aligned without overlap, and all controls are visible and unobstructed." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: text is fully contained, elements are aligned without overlap, and all controls are visible and unobstructed." }1.81 s
banking-ios/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 290, "height": 50 }, "description": "The header title 'Banking' and the brand label above it are cut off at the left edge of the screen, showing 'anking' and a partial 'MAYA' label." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 45, "width": 290, "height": 50 }, "description": "The header title 'Banking' and the brand label above it are cut off at the left edge of the screen, showing 'anking' and a partial 'MAYA' label." }1.91 s
banking-ios/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.70 s
banking-ios/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.92 s
flight-pass-ios/clean/grounding/boarding-pass3éxitoObjetivo alcanzado2.36 s
flight-pass-ios/clean/grounding/check-in3éxitoObjetivo alcanzado2.25 s
flight-pass-ios/clean/grounding/trips-tab3respuesta inválidaObjetivo fallado2.21 s
flight-pass-ios/clean/grounding/profile-tab3éxitoObjetivo alcanzado1.73 s
flight-pass-ios/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: text, cards, button and tab bar are fully visible, aligned and not overlapping or clipped." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: text, cards, button and tab bar are fully visible, aligned and not overlapping or clipped." }2.12 s
flight-pass-ios/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 80 }, "description": "A red 'Flight update' banner overlaps the blue check-in button, covering most of its label so the button text is cut off and the control is partly hidden." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 80 }, "description": "A red 'Flight update' banner overlaps the blue check-in button, covering most of its label so the button text is cut off and the control is partly hidden." }2.08 s
flight-pass-ios/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.84 s
flight-pass-ios/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.54 s
smart-home-android/clean/grounding/more3éxitoObjetivo alcanzado1.58 s
smart-home-android/clean/grounding/room-search3éxitoObjetivo alcanzado1.86 s
smart-home-android/clean/grounding/thermostat-card3éxitoObjetivo alcanzado2.24 s
smart-home-android/clean/grounding/add-device3éxitoObjetivo alcanzado1.75 s
smart-home-android/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, elements do not overlap, and controls are fully visible and unobstructed." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects: text fits within containers, elements do not overlap, and controls are fully visible and unobstructed." }1.97 s
smart-home-android/defect/layout-defect3respuesta inválidaThe answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 525, "width": 636, "height": 96 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its right side is cut off." } · The answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 525, "width": 636, "height": 96 }, "description": "The Kitchen lights list row is shifted right and extends past the right edge of the screen, so it is misaligned with the other rows and its right side is cut off." }2.42 s
smart-home-android/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.67 s
smart-home-android/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.53 s
Gemini 3.8 Flash · 144 ejecuciones de casos

Versiones de modelo devueltas: google/gemini-3.8-flash.

Objetivos con el mismo resultado de acierto/fallo en las tres repeticiones: 24 / 24. La coincidencia también puede significar que un objetivo se falla siempre.

CasoRepeticiónResultadoDesenlaceTiempo
commerce-catalog-desktop/clean/grounding/search1éxitoObjetivo alcanzado2.28 s
commerce-catalog-desktop/clean/grounding/add-product1éxitoObjetivo alcanzado3.52 s
commerce-catalog-desktop/clean/grounding/catalog-query1éxitoObjetivo alcanzado2.57 s
commerce-catalog-desktop/clean/grounding/stock-filter1éxitoObjetivo alcanzado2.31 s
commerce-catalog-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }3.65 s
commerce-catalog-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 67, "height": 68 }, "description": "The page heading text is clipped on the left edge, cutting off 'Products' into 'ducts' and 'MANAGEMENT' into 'IANAGEMENT'." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 67, "height": 68 }, "description": "The page heading text is clipped on the left edge, cutting off 'Products' into 'ducts' and 'MANAGEMENT' into 'IANAGEMENT'." }4.67 s
commerce-catalog-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.46 s
commerce-catalog-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }3.09 s
analytics-dashboard-desktop/clean/grounding/audience-tab1éxitoObjetivo alcanzado2.08 s
analytics-dashboard-desktop/clean/grounding/revenue-card1éxitoObjetivo alcanzado2.17 s
analytics-dashboard-desktop/clean/grounding/search-field1éxitoObjetivo alcanzado2.41 s
analytics-dashboard-desktop/clean/grounding/compare-toggle1éxitoObjetivo alcanzado2.25 s
analytics-dashboard-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering issues detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering issues detected." }6.43 s
analytics-dashboard-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 202, "y": 251, "width": 326, "height": 28 }, "description": "The card heading text is clipped at the right boundary of its parent container." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 202, "y": 251, "width": 326, "height": 28 }, "description": "The card heading text is clipped at the right boundary of its parent container." }4.58 s
analytics-dashboard-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.44 s
analytics-dashboard-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.84 s
iot-control-desktop/clean/grounding/devices-nav1éxitoObjetivo alcanzado2.46 s
iot-control-desktop/clean/grounding/restart-device1éxitoObjetivo alcanzado2.57 s
iot-control-desktop/clean/grounding/auto-mode1éxitoObjetivo alcanzado2.08 s
iot-control-desktop/clean/grounding/pump-card1éxitoObjetivo alcanzado2.26 s
iot-control-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering defects detected; elements are well-aligned, legible, and unclipped." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering defects detected; elements are well-aligned, legible, and unclipped." }12.60 s
iot-control-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The 'Maintenance scheduled' card overlaps the table content and separator lines." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The 'Maintenance scheduled' card overlaps the table content and separator lines." }4.28 s
iot-control-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.41 s
iot-control-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.22 s
banking-ios/clean/grounding/profile1éxitoObjetivo alcanzado10.16 s
banking-ios/clean/grounding/transaction-search1éxitoObjetivo alcanzado2.53 s
banking-ios/clean/grounding/hide-balance1éxitoObjetivo alcanzado1.82 s
banking-ios/clean/grounding/transfer1éxitoObjetivo alcanzado2.23 s
banking-ios/clean/layout-defect1éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 412, "y": 869, "width": 178, "height": 27 }, "description": "Two 'Transfer' text labels are vertically overlapping inside the button, creating a distorted, duplicated text rendering defect." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 412, "y": 869, "width": 178, "height": 27 }, "description": "Two 'Transfer' text labels are vertically overlapping inside the button, creating a distorted, duplicated text rendering defect." }13.11 s
banking-ios/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 59 }, "description": "The header text ('AYA' / 'banking') is cut off by the left viewport edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 59 }, "description": "The header text ('AYA' / 'banking') is cut off by the left viewport edge." }3.90 s
banking-ios/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.90 s
banking-ios/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.58 s
flight-pass-ios/clean/grounding/boarding-pass1éxitoObjetivo alcanzado2.34 s
flight-pass-ios/clean/grounding/check-in1éxitoObjetivo alcanzado2.16 s
flight-pass-ios/clean/grounding/trips-tab1éxitoObjetivo alcanzado1.65 s
flight-pass-ios/clean/grounding/profile-tab1éxitoObjetivo alcanzado1.71 s
flight-pass-ios/clean/layout-defect1éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 405, "y": 866, "width": 189, "height": 26 }, "description": "The \"Check in\" button text is clipped along its bottom edge inside the button container." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 405, "y": 866, "width": 189, "height": 26 }, "description": "The \"Check in\" button text is clipped along its bottom edge inside the button container." }12.77 s
flight-pass-ios/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 45, "y": 850, "width": 910, "height": 68 }, "description": "The red 'Flight update' banner overlaps and obscures the blue button beneath it." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 45, "y": 850, "width": 910, "height": 68 }, "description": "The red 'Flight update' banner overlaps and obscures the blue button beneath it." }4.16 s
flight-pass-ios/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.99 s
flight-pass-ios/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.89 s
smart-home-android/clean/grounding/more1éxitoObjetivo alcanzado2.05 s
smart-home-android/clean/grounding/room-search1éxitoObjetivo alcanzado2.36 s
smart-home-android/clean/grounding/thermostat-card1éxitoObjetivo alcanzado2.22 s
smart-home-android/clean/grounding/add-device1éxitoObjetivo alcanzado1.55 s
smart-home-android/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "None" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "None" }8.97 s
smart-home-android/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 355, "y": 510, "width": 600, "height": 69 }, "description": "The 'Kitchen lights' card is misaligned and unexpectedly indented/broken compared to the other list items in the 'Quick view' section." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 355, "y": 510, "width": 600, "height": 69 }, "description": "The 'Kitchen lights' card is misaligned and unexpectedly indented/broken compared to the other list items in the 'Quick view' section." }7.27 s
smart-home-android/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.35 s
smart-home-android/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.06 s
commerce-catalog-desktop/clean/grounding/search2éxitoObjetivo alcanzado2.53 s
commerce-catalog-desktop/clean/grounding/add-product2éxitoObjetivo alcanzado2.38 s
commerce-catalog-desktop/clean/grounding/catalog-query2éxitoObjetivo alcanzado2.62 s
commerce-catalog-desktop/clean/grounding/stock-filter2éxitoObjetivo alcanzado2.50 s
commerce-catalog-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects detected in the interface." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects detected in the interface." }4.11 s
commerce-catalog-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 68, "height": 73 }, "description": "The page header text ('Products' and category label above it) is horizontally clipped on the left edge by its container boundary." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 68, "height": 73 }, "description": "The page header text ('Products' and category label above it) is horizontally clipped on the left edge by its container boundary." }10.76 s
commerce-catalog-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.12 s
commerce-catalog-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }3.35 s
analytics-dashboard-desktop/clean/grounding/audience-tab2éxitoObjetivo alcanzado8.75 s
analytics-dashboard-desktop/clean/grounding/revenue-card2éxitoObjetivo alcanzado2.04 s
analytics-dashboard-desktop/clean/grounding/search-field2éxitoObjetivo alcanzado1.87 s
analytics-dashboard-desktop/clean/grounding/compare-toggle2éxitoObjetivo alcanzado2.24 s
analytics-dashboard-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual rendering defects detected." }5.95 s
analytics-dashboard-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 224, "width": 339, "height": 142 }, "description": "The heading text inside the first KPI card is cut off and clipped by the card's right boundary." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 224, "width": 339, "height": 142 }, "description": "The heading text inside the first KPI card is cut off and clipped by the card's right boundary." }3.10 s
analytics-dashboard-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.67 s
analytics-dashboard-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.94 s
iot-control-desktop/clean/grounding/devices-nav2éxitoObjetivo alcanzado2.15 s
iot-control-desktop/clean/grounding/restart-device2éxitoObjetivo alcanzado2.22 s
iot-control-desktop/clean/grounding/auto-mode2éxitoObjetivo alcanzado2.25 s
iot-control-desktop/clean/grounding/pump-card2éxitoObjetivo alcanzado2.36 s
iot-control-desktop/clean/layout-defect2éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 185, "y": 118, "width": 227, "height": 70 }, "description": "The subtitle 'FACTORY FLOOR 02' overlaps directly onto the top of the main 'IoT control center' heading." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 185, "y": 118, "width": 227, "height": 70 }, "description": "The subtitle 'FACTORY FLOOR 02' overlaps directly onto the top of the main 'IoT control center' heading." }4.70 s
iot-control-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' badge or overlay is floating awkwardly and overlapping table row divider lines and table contents." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 558, "width": 223, "height": 65 }, "description": "The red 'Maintenance scheduled' badge or overlay is floating awkwardly and overlapping table row divider lines and table contents." }2.14 s
iot-control-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.26 s
iot-control-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.55 s
banking-ios/clean/grounding/profile2éxitoObjetivo alcanzado5.36 s
banking-ios/clean/grounding/transaction-search2éxitoObjetivo alcanzado2.49 s
banking-ios/clean/grounding/hide-balance2éxitoObjetivo alcanzado2.10 s
banking-ios/clean/grounding/transfer2éxitoObjetivo alcanzado2.07 s
banking-ios/clean/layout-defect2éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 413, "y": 868, "width": 174, "height": 22 }, "description": "The 'Transfer' button label has duplicated text overlapping vertically, rendering the text distorted and corrupted." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 413, "y": 868, "width": 174, "height": 22 }, "description": "The 'Transfer' button label has duplicated text overlapping vertically, rendering the text distorted and corrupted." }7.07 s
banking-ios/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 60 }, "description": "The header text in the upper left corner is clipped off by the left viewport edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 60 }, "description": "The header text in the upper left corner is clipped off by the left viewport edge." }4.05 s
banking-ios/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.54 s
banking-ios/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.07 s
flight-pass-ios/clean/grounding/boarding-pass2éxitoObjetivo alcanzado2.06 s
flight-pass-ios/clean/grounding/check-in2éxitoObjetivo alcanzado1.79 s
flight-pass-ios/clean/grounding/trips-tab2éxitoObjetivo alcanzado2.85 s
flight-pass-ios/clean/grounding/profile-tab2éxitoObjetivo alcanzado2.04 s
flight-pass-ios/clean/layout-defect2éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 852, "width": 908, "height": 59 }, "description": "The 'Check in' label inside the primary action button is vertically clipped at the bottom." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 46, "y": 852, "width": 908, "height": 59 }, "description": "The 'Check in' label inside the primary action button is vertically clipped at the bottom." }7.45 s
flight-pass-ios/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 46, "y": 849, "width": 908, "height": 67 }, "description": "The red 'Flight update' notification banner overlaps and obscures the blue button behind it." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 46, "y": 849, "width": 908, "height": 67 }, "description": "The red 'Flight update' notification banner overlaps and obscures the blue button behind it." }2.35 s
flight-pass-ios/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.73 s
flight-pass-ios/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.85 s
smart-home-android/clean/grounding/more2éxitoObjetivo alcanzado2.12 s
smart-home-android/clean/grounding/room-search2éxitoObjetivo alcanzado2.18 s
smart-home-android/clean/grounding/thermostat-card2éxitoObjetivo alcanzado2.07 s
smart-home-android/clean/grounding/add-device2éxitoObjetivo alcanzado2.02 s
smart-home-android/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No rendering defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No rendering defects detected." }6.38 s
smart-home-android/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 509, "width": 600, "height": 71 }, "description": "The second list card ('Kitchen lights') is severely misaligned horizontally compared to the other list items, breaking the expected vertical list layout." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 509, "width": 600, "height": 71 }, "description": "The second list card ('Kitchen lights') is severely misaligned horizontally compared to the other list items, breaking the expected vertical list layout." }8.71 s
smart-home-android/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.21 s
smart-home-android/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.38 s
commerce-catalog-desktop/clean/grounding/search3éxitoObjetivo alcanzado2.32 s
commerce-catalog-desktop/clean/grounding/add-product3éxitoObjetivo alcanzado2.40 s
commerce-catalog-desktop/clean/grounding/catalog-query3éxitoObjetivo alcanzado2.62 s
commerce-catalog-desktop/clean/grounding/stock-filter3éxitoObjetivo alcanzado3.04 s
commerce-catalog-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible defects detected." }10.99 s
commerce-catalog-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 127, "width": 68, "height": 67 }, "description": "The section title text ('ducts') and subtitle ('IANAGEMENT') are clipped on the left edge by the sidebar/container boundary." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 127, "width": 68, "height": 67 }, "description": "The section title text ('ducts') and subtitle ('IANAGEMENT') are clipped on the left edge by the sidebar/container boundary." }2.32 s
commerce-catalog-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.51 s
commerce-catalog-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.85 s
analytics-dashboard-desktop/clean/grounding/audience-tab3éxitoObjetivo alcanzado9.88 s
analytics-dashboard-desktop/clean/grounding/revenue-card3éxitoObjetivo alcanzado2.26 s
analytics-dashboard-desktop/clean/grounding/search-field3éxitoObjetivo alcanzado2.06 s
analytics-dashboard-desktop/clean/grounding/compare-toggle3éxitoObjetivo alcanzado2.07 s
analytics-dashboard-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual defects detected in the interface." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visual defects detected in the interface." }5.59 s
analytics-dashboard-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 187, "y": 223, "width": 340, "height": 143 }, "description": "The metric card title 'Monthly revenue across all sales chann' is clipped by the container's right boundary." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 187, "y": 223, "width": 340, "height": 143 }, "description": "The metric card title 'Monthly revenue across all sales chann' is clipped by the container's right boundary." }5.54 s
analytics-dashboard-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.80 s
analytics-dashboard-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.89 s
iot-control-desktop/clean/grounding/devices-nav3éxitoObjetivo alcanzado2.31 s
iot-control-desktop/clean/grounding/restart-device3éxitoObjetivo alcanzado2.36 s
iot-control-desktop/clean/grounding/auto-mode3éxitoObjetivo alcanzado2.29 s
iot-control-desktop/clean/grounding/pump-card3éxitoObjetivo alcanzado2.74 s
iot-control-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects detected in the interface." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects detected in the interface." }11.74 s
iot-control-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 223, "height": 67 }, "description": "The 'Maintenance scheduled' banner overlaps directly on top of the data table and its row borders." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 223, "height": 67 }, "description": "The 'Maintenance scheduled' banner overlaps directly on top of the data table and its row borders." }13.83 s
iot-control-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.37 s
iot-control-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.58 s
banking-ios/clean/grounding/profile3éxitoObjetivo alcanzado12.22 s
banking-ios/clean/grounding/transaction-search3éxitoObjetivo alcanzado3.27 s
banking-ios/clean/grounding/hide-balance3éxitoObjetivo alcanzado2.09 s
banking-ios/clean/grounding/transfer3éxitoObjetivo alcanzado1.99 s
banking-ios/clean/layout-defect3éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 410, "y": 866, "width": 180, "height": 25 }, "description": "Two text labels within the 'Transfer' button overlap directly on top of each other, creating garbled and obscured text." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 410, "y": 866, "width": 180, "height": 25 }, "description": "Two text labels within the 'Transfer' button overlap directly on top of each other, creating garbled and obscured text." }8.34 s
banking-ios/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 62 }, "description": "The title text 'banking' and the header label above it are cropped by the left edge of the viewport." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 284, "height": 62 }, "description": "The title text 'banking' and the header label above it are cropped by the left edge of the viewport." }3.69 s
banking-ios/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.23 s
banking-ios/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.50 s
flight-pass-ios/clean/grounding/boarding-pass3éxitoObjetivo alcanzado2.05 s
flight-pass-ios/clean/grounding/check-in3éxitoObjetivo alcanzado2.25 s
flight-pass-ios/clean/grounding/trips-tab3éxitoObjetivo alcanzado2.51 s
flight-pass-ios/clean/grounding/profile-tab3éxitoObjetivo alcanzado1.92 s
flight-pass-ios/clean/layout-defect3respuesta inválidaThe answer is not valid JSON. Parsing failed before schema validation could run. JSON parsing error: Unterminated string in JSON at position 213 (line 1 column 214) Malformed answers are not repaired in this pilot and count as incorrect, even when the intended defect category is recognizable. Expected: { "has_issue": false, "category": null, "region": null } Returned: {"has_issue": true, "category": "container_clipping", "region": {"x": 409, "y": 869, "width": 182, "height": 22}, "description": "The bottom portion of the 'Check in' button text is clipped horizontally inside its · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: {"has_issue": true, "category": "container_clipping", "region": {"x": 409, "y": 869, "width": 182, "height": 22}, "description": "The bottom portion of the 'Check in' button text is clipped horizontally inside its17.47 s
flight-pass-ios/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 66 }, "description": "A red 'Flight update' notification banner overlaps and obscures the blue 'Check in' action button." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 66 }, "description": "A red 'Flight update' notification banner overlaps and obscures the blue 'Check in' action button." }4.31 s
flight-pass-ios/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.41 s
flight-pass-ios/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.49 s
smart-home-android/clean/grounding/more3éxitoObjetivo alcanzado1.79 s
smart-home-android/clean/grounding/room-search3éxitoObjetivo alcanzado2.07 s
smart-home-android/clean/grounding/thermostat-card3éxitoObjetivo alcanzado2.30 s
smart-home-android/clean/grounding/add-device3éxitoObjetivo alcanzado2.25 s
smart-home-android/clean/layout-defect3éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 140, "y": 966, "width": 725, "height": 34 }, "description": "The bottom navigation bar labels ('Home', 'Rooms', 'Activity') are partially cut off by the bottom viewport edge." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 140, "y": 966, "width": 725, "height": 34 }, "description": "The bottom navigation bar labels ('Home', 'Rooms', 'Activity') are partially cut off by the bottom viewport edge." }8.35 s
smart-home-android/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 355, "y": 510, "width": 600, "height": 70 }, "description": "The 'Kitchen lights' list item card is significantly shifted and misaligned from the vertical grid layout used by surrounding items." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 355, "y": 510, "width": 600, "height": 70 }, "description": "The 'Kitchen lights' list item card is significantly shifted and misaligned from the vertical grid layout used by surrounding items." }8.83 s
smart-home-android/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.84 s
smart-home-android/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.62 s
Grok 4.7 · 144 ejecuciones de casos

Versiones de modelo devueltas: x-ai/grok-4.7.

Objetivos con el mismo resultado de acierto/fallo en las tres repeticiones: 16 / 24. La coincidencia también puede significar que un objetivo se falla siempre.

CasoRepeticiónResultadoDesenlaceTiempo
commerce-catalog-desktop/clean/grounding/search1éxitoObjetivo alcanzado12.82 s
commerce-catalog-desktop/clean/grounding/add-product1éxitoObjetivo alcanzado7.11 s
commerce-catalog-desktop/clean/grounding/catalog-query1éxitoObjetivo alcanzado9.00 s
commerce-catalog-desktop/clean/grounding/stock-filter1éxitoObjetivo alcanzado28.50 s
commerce-catalog-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect." }12.36 s
commerce-catalog-desktop/defect/layout-defect1respuesta inválidaThe answer is not valid JSON. Parsing failed before schema validation could run. JSON parsing error: Unterminated string in JSON at position 195 (line 1 column 196) Malformed answers are not repaired in this pilot and count as incorrect, even when the intended defect category is recognizable. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: {"has_issue":true,"category":"container_clipping","region":{"x":278,"y":114,"width":99,"height":65},"description":"The page heading is clipped on the left, showing only “IANAGEMENT” and “ducts.”} · The answer is not valid JSON. Parsing failed before schema validation could run. JSON parsing error: Unterminated string in JSON at position 195 (line 1 column 196) Malformed answers are not repaired in this pilot and count as incorrect, even when the intended defect category is recognizable. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: {"has_issue":true,"category":"container_clipping","region":{"x":278,"y":114,"width":99,"height":65},"description":"The page heading is clipped on the left, showing only “IANAGEMENT” and “ducts.”}3.06 s
commerce-catalog-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.39 s
commerce-catalog-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.97 s
analytics-dashboard-desktop/clean/grounding/audience-tab1éxitoObjetivo alcanzado26.90 s
analytics-dashboard-desktop/clean/grounding/revenue-card1éxitoObjetivo alcanzado9.62 s
analytics-dashboard-desktop/clean/grounding/search-field1éxitoObjetivo alcanzado11.49 s
analytics-dashboard-desktop/clean/grounding/compare-toggle1éxitoObjetivo fallado27.51 s
analytics-dashboard-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, overflow, overlap, obscured control, or broken reflow is visible." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, overflow, overlap, obscured control, or broken reflow is visible." }92.19 s
analytics-dashboard-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 204, "y": 226, "width": 318, "height": 34 }, "description": "The revenue card title is truncated mid-word at the card's right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 204, "y": 226, "width": 318, "height": 34 }, "description": "The revenue card title is truncated mid-word at the card's right edge." }18.34 s
analytics-dashboard-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.54 s
analytics-dashboard-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }18.39 s
iot-control-desktop/clean/grounding/devices-nav1éxitoObjetivo alcanzado13.27 s
iot-control-desktop/clean/grounding/restart-device1éxitoObjetivo alcanzado15.28 s
iot-control-desktop/clean/grounding/auto-mode1éxitoObjetivo alcanzado25.79 s
iot-control-desktop/clean/grounding/pump-card1éxitoObjetivo alcanzado14.44 s
iot-control-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect; all text, controls, cards, and table rows are fully contained and aligned." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect; all text, controls, cards, and table rows are fully contained and aligned." }6.73 s
iot-control-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 556, "width": 219, "height": 66 }, "description": "A red 'Maintenance scheduled' banner overlaps the device table rows." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 556, "width": 219, "height": 66 }, "description": "A red 'Maintenance scheduled' banner overlaps the device table rows." }25.49 s
iot-control-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.88 s
iot-control-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.87 s
banking-ios/clean/grounding/profile1éxitoObjetivo fallado21.93 s
banking-ios/clean/grounding/transaction-search1éxitoObjetivo alcanzado10.19 s
banking-ios/clean/grounding/hide-balance1éxitoObjetivo alcanzado54.00 s
banking-ios/clean/grounding/transfer1éxitoObjetivo alcanzado11.31 s
banking-ios/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the listed categories." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the listed categories." }8.19 s
banking-ios/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 224, "height": 34 }, "description": "The 'Banking' heading is cropped by the left viewport edge, leaving only 'anking' visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 48, "width": 224, "height": 34 }, "description": "The 'Banking' heading is cropped by the left viewport edge, leaving only 'anking' visible." }20.63 s
banking-ios/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.66 s
banking-ios/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.00 s
flight-pass-ios/clean/grounding/boarding-pass1éxitoObjetivo alcanzado14.68 s
flight-pass-ios/clean/grounding/check-in1éxitoObjetivo alcanzado8.91 s
flight-pass-ios/clean/grounding/trips-tab1éxitoObjetivo alcanzado17.82 s
flight-pass-ios/clean/grounding/profile-tab1éxitoObjetivo alcanzado17.09 s
flight-pass-ios/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }12.00 s
flight-pass-ios/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 48, "y": 842, "width": 904, "height": 90 }, "description": "Red Flight update banner overlaps a blue banner, obscuring its text." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 48, "y": 842, "width": 904, "height": 90 }, "description": "Red Flight update banner overlaps a blue banner, obscuring its text." }19.37 s
flight-pass-ios/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.86 s
flight-pass-ios/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.58 s
smart-home-android/clean/grounding/more1éxitoObjetivo fallado8.75 s
smart-home-android/clean/grounding/room-search1éxitoObjetivo alcanzado9.33 s
smart-home-android/clean/grounding/thermostat-card1éxitoObjetivo alcanzado13.72 s
smart-home-android/clean/grounding/add-device1éxitoObjetivo fallado8.84 s
smart-home-android/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }3.17 s
smart-home-android/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 366, "y": 509, "width": 588, "height": 83 }, "description": "The Kitchen lights card is shifted right and does not align with the full-width cards above and below it." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 366, "y": 509, "width": 588, "height": 83 }, "description": "The Kitchen lights card is shifted right and does not align with the full-width cards above and below it." }20.28 s
smart-home-android/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.88 s
smart-home-android/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.52 s
commerce-catalog-desktop/clean/grounding/search2éxitoObjetivo alcanzado26.80 s
commerce-catalog-desktop/clean/grounding/add-product2éxitoObjetivo alcanzado7.36 s
commerce-catalog-desktop/clean/grounding/catalog-query2éxitoObjetivo alcanzado20.98 s
commerce-catalog-desktop/clean/grounding/stock-filter2éxitoObjetivo fallado17.11 s
commerce-catalog-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }5.63 s
commerce-catalog-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 276, "y": 112, "width": 103, "height": 68 }, "description": "The page heading is clipped at its left edge, cutting off the beginning of both heading lines." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 276, "y": 112, "width": 103, "height": 68 }, "description": "The page heading is clipped at its left edge, cutting off the beginning of both heading lines." }3.07 s
commerce-catalog-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.73 s
commerce-catalog-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.08 s
analytics-dashboard-desktop/clean/grounding/audience-tab2éxitoObjetivo alcanzado18.38 s
analytics-dashboard-desktop/clean/grounding/revenue-card2éxitoObjetivo alcanzado16.61 s
analytics-dashboard-desktop/clean/grounding/search-field2éxitoObjetivo fallado38.77 s
analytics-dashboard-desktop/clean/grounding/compare-toggle2éxitoObjetivo fallado21.24 s
analytics-dashboard-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect." }45.43 s
analytics-dashboard-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 228, "width": 324, "height": 46 }, "description": "The metric card heading is cut off mid-word at the card's right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 228, "width": 324, "height": 46 }, "description": "The metric card heading is cut off mid-word at the card's right edge." }14.96 s
analytics-dashboard-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.22 s
analytics-dashboard-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.97 s
iot-control-desktop/clean/grounding/devices-nav2éxitoObjetivo alcanzado21.10 s
iot-control-desktop/clean/grounding/restart-device2éxitoObjetivo alcanzado11.53 s
iot-control-desktop/clean/grounding/auto-mode2éxitoObjetivo fallado16.62 s
iot-control-desktop/clean/grounding/pump-card2éxitoObjetivo alcanzado22.31 s
iot-control-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects." }13.76 s
iot-control-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 458, "y": 555, "width": 224, "height": 69 }, "description": "A red \"Maintenance scheduled\" banner overlaps and obscures the device table rows." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 458, "y": 555, "width": 224, "height": 69 }, "description": "A red \"Maintenance scheduled\" banner overlaps and obscures the device table rows." }12.82 s
iot-control-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.78 s
iot-control-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.89 s
banking-ios/clean/grounding/profile2éxitoObjetivo fallado20.19 s
banking-ios/clean/grounding/transaction-search2éxitoObjetivo alcanzado24.96 s
banking-ios/clean/grounding/hide-balance2éxitoObjetivo fallado8.56 s
banking-ios/clean/grounding/transfer2éxitoObjetivo alcanzado8.18 s
banking-ios/clean/layout-defect2éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 58, "y": 962, "width": 814, "height": 38 }, "description": "Bottom navigation labels are cut off by the lower viewport edge." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 58, "y": 962, "width": 814, "height": 38 }, "description": "Bottom navigation labels are cut off by the lower viewport edge." }18.85 s
banking-ios/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 58, "width": 500, "height": 48 }, "description": "The 'Banking' heading is cropped at the left viewport edge, with the leading 'B' cut off." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 58, "width": 500, "height": 48 }, "description": "The 'Banking' heading is cropped at the left viewport edge, with the leading 'B' cut off." }20.32 s
banking-ios/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.54 s
banking-ios/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.83 s
flight-pass-ios/clean/grounding/boarding-pass2éxitoObjetivo alcanzado12.05 s
flight-pass-ios/clean/grounding/check-in2éxitoObjetivo fallado15.24 s
flight-pass-ios/clean/grounding/trips-tab2éxitoObjetivo alcanzado23.46 s
flight-pass-ios/clean/grounding/profile-tab2éxitoObjetivo alcanzado25.79 s
flight-pass-ios/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect; content fits the viewport without clipping, overflow, overlap, or broken reflow." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect; content fits the viewport without clipping, overflow, overlap, or broken reflow." }7.64 s
flight-pass-ios/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 42, "y": 812, "width": 918, "height": 92 }, "description": "A red flight-update banner overlaps a blue card behind it, obscuring that card's text." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 42, "y": 812, "width": 918, "height": 92 }, "description": "A red flight-update banner overlaps a blue card behind it, obscuring that card's text." }12.45 s
flight-pass-ios/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.96 s
flight-pass-ios/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.77 s
smart-home-android/clean/grounding/more2éxitoObjetivo fallado16.25 s
smart-home-android/clean/grounding/room-search2éxitoObjetivo alcanzado16.47 s
smart-home-android/clean/grounding/thermostat-card2éxitoObjetivo alcanzado19.16 s
smart-home-android/clean/grounding/add-device2éxitoObjetivo alcanzado9.97 s
smart-home-android/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect." }15.40 s
smart-home-android/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 328, "y": 567, "width": 632, "height": 110 }, "description": "The Kitchen lights card is shifted right and overlaps the Front door card beneath it." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 328, "y": 567, "width": 632, "height": 110 }, "description": "The Kitchen lights card is shifted right and overlaps the Front door card beneath it." }40.90 s
smart-home-android/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.51 s
smart-home-android/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }5.93 s
commerce-catalog-desktop/clean/grounding/search3éxitoObjetivo alcanzado27.72 s
commerce-catalog-desktop/clean/grounding/add-product3respuesta inválidaObjetivo fallado11.21 s
commerce-catalog-desktop/clean/grounding/catalog-query3éxitoObjetivo alcanzado21.57 s
commerce-catalog-desktop/clean/grounding/stock-filter3éxitoObjetivo fallado17.41 s
commerce-catalog-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, overflow, overlap, obscured control, or broken reflow is visible." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No viewport cropping, clipping, overflow, overlap, obscured control, or broken reflow is visible." }13.76 s
commerce-catalog-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 273, "y": 116, "width": 102, "height": 62 }, "description": "Heading text is clipped at its left edge, showing incomplete words." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 273, "y": 116, "width": 102, "height": 62 }, "description": "Heading text is clipped at its left edge, showing incomplete words." }3.93 s
commerce-catalog-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.19 s
commerce-catalog-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.49 s
analytics-dashboard-desktop/clean/grounding/audience-tab3éxitoObjetivo alcanzado24.18 s
analytics-dashboard-desktop/clean/grounding/revenue-card3éxitoObjetivo alcanzado9.67 s
analytics-dashboard-desktop/clean/grounding/search-field3éxitoObjetivo alcanzado6.95 s
analytics-dashboard-desktop/clean/grounding/compare-toggle3éxitoObjetivo fallado10.42 s
analytics-dashboard-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the listed categories." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the listed categories." }35.63 s
analytics-dashboard-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 294, "y": 224, "width": 459, "height": 34 }, "description": "The revenue card heading is cut off at the card's right edge, truncating the title text." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 294, "y": 224, "width": 459, "height": 34 }, "description": "The revenue card heading is cut off at the card's right edge, truncating the title text." }18.82 s
analytics-dashboard-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }6.03 s
analytics-dashboard-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.56 s
iot-control-desktop/clean/grounding/devices-nav3éxitoObjetivo alcanzado13.95 s
iot-control-desktop/clean/grounding/restart-device3éxitoObjetivo alcanzado8.91 s
iot-control-desktop/clean/grounding/auto-mode3éxitoObjetivo alcanzado18.87 s
iot-control-desktop/clean/grounding/pump-card3éxitoObjetivo alcanzado19.79 s
iot-control-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect; text, cards, table, and controls render within their containers." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defect; text, cards, table, and controls render within their containers." }13.84 s
iot-control-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 463, "y": 556, "width": 223, "height": 68 }, "description": "A red maintenance banner overlaps the device table, covering the Cooling pump status and row boundary." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 463, "y": 556, "width": 223, "height": 68 }, "description": "A red maintenance banner overlaps the device table, covering the Cooling pump status and row boundary." }5.51 s
iot-control-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.98 s
iot-control-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.08 s
banking-ios/clean/grounding/profile3éxitoObjetivo fallado17.73 s
banking-ios/clean/grounding/transaction-search3éxitoObjetivo alcanzado17.27 s
banking-ios/clean/grounding/hide-balance3éxitoObjetivo fallado21.90 s
banking-ios/clean/grounding/transfer3éxitoObjetivo alcanzado7.91 s
banking-ios/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }9.43 s
banking-ios/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 49, "width": 162, "height": 44 }, "description": "The Banking header title is cropped at the left viewport edge, leaving only anking visible." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 49, "width": 162, "height": 44 }, "description": "The Banking header title is cropped at the left viewport edge, leaving only anking visible." }12.40 s
banking-ios/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.41 s
banking-ios/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.02 s
flight-pass-ios/clean/grounding/boarding-pass3éxitoObjetivo alcanzado12.12 s
flight-pass-ios/clean/grounding/check-in3éxitoObjetivo fallado17.61 s
flight-pass-ios/clean/grounding/trips-tab3éxitoObjetivo alcanzado18.95 s
flight-pass-ios/clean/grounding/profile-tab3éxitoObjetivo alcanzado32.81 s
flight-pass-ios/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect in the specified categories." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defect in the specified categories." }10.62 s
flight-pass-ios/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 41, "y": 829, "width": 877, "height": 90 }, "description": "The red Flight update banner overlaps the blue banner behind it, obscuring and clipping its text." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 41, "y": 829, "width": 877, "height": 90 }, "description": "The red Flight update banner overlaps the blue banner behind it, obscuring and clipping its text." }17.93 s
flight-pass-ios/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.46 s
flight-pass-ios/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.66 s
smart-home-android/clean/grounding/more3éxitoObjetivo fallado4.33 s
smart-home-android/clean/grounding/room-search3éxitoObjetivo alcanzado6.48 s
smart-home-android/clean/grounding/thermostat-card3éxitoObjetivo fallado22.65 s
smart-home-android/clean/grounding/add-device3éxitoObjetivo alcanzado9.77 s
smart-home-android/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the specified categories." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects in the specified categories." }10.69 s
smart-home-android/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 430, "y": 510, "width": 528, "height": 80 }, "description": "The Kitchen lights card is indented and narrower than the full-width sibling cards, breaking the list reflow." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 430, "y": 510, "width": 528, "height": 80 }, "description": "The Kitchen lights card is indented and narrower than the full-width sibling cards, breaking the list reflow." }23.64 s
smart-home-android/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.57 s
smart-home-android/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.55 s
Qwen3.8 Max (0902) · 144 ejecuciones de casos

Versiones de modelo devueltas: qwen/qwen3.8-max-0902.

Objetivos con el mismo resultado de acierto/fallo en las tres repeticiones: 24 / 24. La coincidencia también puede significar que un objetivo se falla siempre.

CasoRepeticiónResultadoDesenlaceTiempo
commerce-catalog-desktop/clean/grounding/search1éxitoObjetivo alcanzado6.70 s
commerce-catalog-desktop/clean/grounding/add-product1éxitoObjetivo alcanzado9.03 s
commerce-catalog-desktop/clean/grounding/catalog-query1éxitoObjetivo alcanzado9.96 s
commerce-catalog-desktop/clean/grounding/stock-filter1éxitoObjetivo alcanzado7.91 s
commerce-catalog-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." }11.01 s
commerce-catalog-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 190, "y": 120, "width": 180, "height": 75 }, "description": "Page heading is cut off at the left edge, showing 'ducts' instead of 'Products' and 'IANAGEMENT' instead of 'MANAGEMENT'." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 190, "y": 120, "width": 180, "height": 75 }, "description": "Page heading is cut off at the left edge, showing 'ducts' instead of 'Products' and 'IANAGEMENT' instead of 'MANAGEMENT'." }15.27 s
commerce-catalog-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }4.24 s
commerce-catalog-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }5.16 s
analytics-dashboard-desktop/clean/grounding/audience-tab1éxitoObjetivo alcanzado14.38 s
analytics-dashboard-desktop/clean/grounding/revenue-card1éxitoObjetivo alcanzado11.24 s
analytics-dashboard-desktop/clean/grounding/search-field1éxitoObjetivo alcanzado10.90 s
analytics-dashboard-desktop/clean/grounding/compare-toggle1éxitoObjetivo alcanzado11.42 s
analytics-dashboard-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." }11.64 s
analytics-dashboard-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 200, "y": 245, "width": 330, "height": 40 }, "description": "Card title 'Monthly revenue across all sales channels' is cut off at the card's right edge, truncating the word 'channels'." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 200, "y": 245, "width": 330, "height": 40 }, "description": "Card title 'Monthly revenue across all sales channels' is cut off at the card's right edge, truncating the word 'channels'." }14.56 s
analytics-dashboard-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }4.22 s
analytics-dashboard-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }5.52 s
iot-control-desktop/clean/grounding/devices-nav1éxitoObjetivo alcanzado11.78 s
iot-control-desktop/clean/grounding/restart-device1éxitoObjetivo alcanzado11.50 s
iot-control-desktop/clean/grounding/auto-mode1éxitoObjetivo alcanzado14.37 s
iot-control-desktop/clean/grounding/pump-card1éxitoObjetivo alcanzado8.43 s
iot-control-desktop/clean/layout-defect1respuesta inválidaThe answer did not satisfy the response schema and counts as incorrect. Schema errors: [{"instancePath":"/description","schemaPath":"#/properties/description/maxLength","keyword":"maxLength","params":{"limit":160},"message":"must NOT have more than 160 characters"}] Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: cards, table, toggle, and button render fully within their containers without cropping, clipping, overflow, overlap, or obscured controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: cards, table, toggle, and button render fully within their containers without cropping, clipping, overflow, overlap, or obscured controls." }10.94 s
iot-control-desktop/defect/layout-defect1éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; toast notification overlays content intentionally without obscuring actionable controls." } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; toast notification overlays content intentionally without obscuring actionable controls." }14.07 s
iot-control-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.83 s
iot-control-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }6.18 s
banking-ios/clean/grounding/profile1éxitoObjetivo alcanzado9.52 s
banking-ios/clean/grounding/transaction-search1éxitoObjetivo alcanzado6.30 s
banking-ios/clean/grounding/hide-balance1éxitoObjetivo alcanzado18.69 s
banking-ios/clean/grounding/transfer1éxitoObjetivo alcanzado11.96 s
banking-ios/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all containers, text, and controls render fully within the viewport without clipping, overflow, or overlap." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all containers, text, and controls render fully within the viewport without clipping, overflow, or overlap." }9.44 s
banking-ios/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 280, "height": 60 }, "description": "Page heading 'Banking' is clipped at the left edge; the leading 'B' is cut off by the viewport." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 280, "height": 60 }, "description": "Page heading 'Banking' is clipped at the left edge; the leading 'B' is cut off by the viewport." }9.56 s
banking-ios/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.62 s
banking-ios/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }4.11 s
flight-pass-ios/clean/grounding/boarding-pass1éxitoObjetivo alcanzado8.08 s
flight-pass-ios/clean/grounding/check-in1éxitoObjetivo alcanzado6.42 s
flight-pass-ios/clean/grounding/trips-tab1éxitoObjetivo alcanzado8.58 s
flight-pass-ios/clean/grounding/profile-tab1éxitoObjetivo alcanzado7.65 s
flight-pass-ios/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." }8.78 s
flight-pass-ios/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 855, "width": 920, "height": 65 }, "description": "Red 'Flight update' toast banner overlaps and obscures the blue 'Check in' button text beneath it." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 855, "width": 920, "height": 65 }, "description": "Red 'Flight update' toast banner overlaps and obscures the blue 'Check in' button text beneath it." }14.78 s
flight-pass-ios/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.73 s
flight-pass-ios/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }4.51 s
smart-home-android/clean/grounding/more1éxitoObjetivo alcanzado9.39 s
smart-home-android/clean/grounding/room-search1éxitoObjetivo alcanzado5.70 s
smart-home-android/clean/grounding/thermostat-card1éxitoObjetivo alcanzado7.16 s
smart-home-android/clean/grounding/add-device1éxitoObjetivo alcanzado6.43 s
smart-home-android/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without overlap or clipping." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without overlap or clipping." }9.54 s
smart-home-android/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 355, "y": 510, "width": 645, "height": 70 }, "description": "The 'Kitchen lights' card (item 2) is shifted right and clipped at the right viewport edge, cutting off its border and chevron control." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 355, "y": 510, "width": 645, "height": 70 }, "description": "The 'Kitchen lights' card (item 2) is shifted right and clipped at the right viewport edge, cutting off its border and chevron control." }14.85 s
smart-home-android/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.91 s
smart-home-android/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }4.46 s
commerce-catalog-desktop/clean/grounding/search2éxitoObjetivo alcanzado7.87 s
commerce-catalog-desktop/clean/grounding/add-product2éxitoObjetivo alcanzado10.41 s
commerce-catalog-desktop/clean/grounding/catalog-query2éxitoObjetivo alcanzado10.06 s
commerce-catalog-desktop/clean/grounding/stock-filter2éxitoObjetivo alcanzado12.58 s
commerce-catalog-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all content fits within containers and viewport." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all content fits within containers and viewport." }8.64 s
commerce-catalog-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 190, "y": 120, "width": 180, "height": 75 }, "description": "Page heading 'Products' is clipped on the left, showing 'ducts', and the 'MANAGEMENT' eyebrow label is cut off at the container edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 190, "y": 120, "width": 180, "height": 75 }, "description": "Page heading 'Products' is clipped on the left, showing 'ducts', and the 'MANAGEMENT' eyebrow label is cut off at the container edge." }13.24 s
commerce-catalog-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.67 s
commerce-catalog-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }5.86 s
analytics-dashboard-desktop/clean/grounding/audience-tab2éxitoObjetivo alcanzado11.26 s
analytics-dashboard-desktop/clean/grounding/revenue-card2éxitoObjetivo alcanzado9.96 s
analytics-dashboard-desktop/clean/grounding/search-field2éxitoObjetivo alcanzado9.24 s
analytics-dashboard-desktop/clean/grounding/compare-toggle2éxitoObjetivo alcanzado12.36 s
analytics-dashboard-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all content fits within containers and viewport." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all content fits within containers and viewport." }10.56 s
analytics-dashboard-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 245, "width": 340, "height": 40 }, "description": "Card title 'Monthly revenue across all sales channels' is truncated at the right edge of its container, cutting off the final letters." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 188, "y": 245, "width": 340, "height": 40 }, "description": "Card title 'Monthly revenue across all sales channels' is truncated at the right edge of its container, cutting off the final letters." }13.44 s
analytics-dashboard-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }4.18 s
analytics-dashboard-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }5.57 s
iot-control-desktop/clean/grounding/devices-nav2éxitoObjetivo alcanzado6.57 s
iot-control-desktop/clean/grounding/restart-device2éxitoObjetivo alcanzado8.10 s
iot-control-desktop/clean/grounding/auto-mode2éxitoObjetivo alcanzado10.17 s
iot-control-desktop/clean/grounding/pump-card2éxitoObjetivo alcanzado5.96 s
iot-control-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without overlap or clipping." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without overlap or clipping." }9.94 s
iot-control-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 223, "height": 63 }, "description": "A red 'Maintenance scheduled' toast/banner overlays the device table rows, covering the area between the Cooling pump and Intake valve entries." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 223, "height": 63 }, "description": "A red 'Maintenance scheduled' toast/banner overlays the device table rows, covering the area between the Cooling pump and Intake valve entries." }22.10 s
iot-control-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.75 s
iot-control-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }5.03 s
banking-ios/clean/grounding/profile2éxitoObjetivo alcanzado8.60 s
banking-ios/clean/grounding/transaction-search2éxitoObjetivo alcanzado8.13 s
banking-ios/clean/grounding/hide-balance2éxitoObjetivo alcanzado6.74 s
banking-ios/clean/grounding/transfer2éxitoObjetivo alcanzado6.15 s
banking-ios/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." }8.60 s
banking-ios/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 300, "height": 60 }, "description": "Page title 'Banking' is cut off at the left viewport edge, showing only 'anking'." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 300, "height": 60 }, "description": "Page title 'Banking' is cut off at the left viewport edge, showing only 'anking'." }11.66 s
banking-ios/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.97 s
banking-ios/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }6.69 s
flight-pass-ios/clean/grounding/boarding-pass2éxitoObjetivo alcanzado10.94 s
flight-pass-ios/clean/grounding/check-in2éxitoObjetivo alcanzado6.76 s
flight-pass-ios/clean/grounding/trips-tab2éxitoObjetivo alcanzado7.43 s
flight-pass-ios/clean/grounding/profile-tab2éxitoObjetivo alcanzado7.58 s
flight-pass-ios/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all text fits, controls are unobscured, and elements are properly spaced." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all text fits, controls are unobscured, and elements are properly spaced." }9.35 s
flight-pass-ios/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 920, "height": 70 }, "description": "The red 'Flight update' toast banner overlaps and obscures the blue 'Check in' button text beneath it." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 920, "height": 70 }, "description": "The red 'Flight update' toast banner overlaps and obscures the blue 'Check in' button text beneath it." }11.78 s
flight-pass-ios/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.27 s
flight-pass-ios/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }5.39 s
smart-home-android/clean/grounding/more2éxitoObjetivo alcanzado6.06 s
smart-home-android/clean/grounding/room-search2éxitoObjetivo alcanzado6.43 s
smart-home-android/clean/grounding/thermostat-card2éxitoObjetivo alcanzado6.58 s
smart-home-android/clean/grounding/add-device2éxitoObjetivo alcanzado6.15 s
smart-home-android/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without cropping, clipping, overflow, overlap, or obscured controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without cropping, clipping, overflow, overlap, or obscured controls." }9.39 s
smart-home-android/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 350, "y": 510, "width": 650, "height": 75 }, "description": "The 'Kitchen lights' quick-view card is offset to the right and clipped by the right viewport edge, breaking the list's alignment." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 350, "y": 510, "width": 650, "height": 75 }, "description": "The 'Kitchen lights' quick-view card is offset to the right and clipped by the right viewport edge, breaking the list's alignment." }17.60 s
smart-home-android/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }4.70 s
smart-home-android/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }6.14 s
commerce-catalog-desktop/clean/grounding/search3éxitoObjetivo alcanzado9.29 s
commerce-catalog-desktop/clean/grounding/add-product3éxitoObjetivo alcanzado10.12 s
commerce-catalog-desktop/clean/grounding/catalog-query3éxitoObjetivo alcanzado9.57 s
commerce-catalog-desktop/clean/grounding/stock-filter3éxitoObjetivo alcanzado5.48 s
commerce-catalog-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within their containers and the viewport." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within their containers and the viewport." }7.69 s
commerce-catalog-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 165, "y": 110, "width": 120, "height": 70 }, "description": "Page heading is cut off on the left: 'MANAGEMENT' appears as 'IANAGEMENT' and 'Products' as 'ducts', clipped by the sidebar edge." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 165, "y": 110, "width": 120, "height": 70 }, "description": "Page heading is cut off on the left: 'MANAGEMENT' appears as 'IANAGEMENT' and 'Products' as 'ducts', clipped by the sidebar edge." }16.10 s
commerce-catalog-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }4.78 s
commerce-catalog-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }8.14 s
analytics-dashboard-desktop/clean/grounding/audience-tab3éxitoObjetivo alcanzado7.93 s
analytics-dashboard-desktop/clean/grounding/revenue-card3éxitoObjetivo alcanzado12.24 s
analytics-dashboard-desktop/clean/grounding/search-field3éxitoObjetivo alcanzado15.80 s
analytics-dashboard-desktop/clean/grounding/compare-toggle3éxitoObjetivo alcanzado11.01 s
analytics-dashboard-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." }10.27 s
analytics-dashboard-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 248, "width": 328, "height": 32 }, "description": "Card title \"Monthly revenue across all sales chann\" is truncated at the card's right edge, cutting off the word \"channels\"." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 248, "width": 328, "height": 32 }, "description": "Card title \"Monthly revenue across all sales chann\" is truncated at the card's right edge, cutting off the word \"channels\"." }12.46 s
analytics-dashboard-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }7.10 s
analytics-dashboard-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }4.95 s
iot-control-desktop/clean/grounding/devices-nav3éxitoObjetivo alcanzado5.90 s
iot-control-desktop/clean/grounding/restart-device3éxitoObjetivo alcanzado8.96 s
iot-control-desktop/clean/grounding/auto-mode3éxitoObjetivo alcanzado7.81 s
iot-control-desktop/clean/grounding/pump-card3éxitoObjetivo alcanzado8.01 s
iot-control-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects; layout is intact with no cropping, clipping, overflow, overlap, or obscured controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible rendering defects; layout is intact with no cropping, clipping, overflow, overlap, or obscured controls." }10.18 s
iot-control-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 222, "height": 65 }, "description": "A red 'Maintenance scheduled' toast/banner overlaps the device table rows, covering content between the Cooling pump and Intake valve rows." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 462, "y": 557, "width": 222, "height": 65 }, "description": "A red 'Maintenance scheduled' toast/banner overlaps the device table rows, covering content between the Cooling pump and Intake valve rows." }11.95 s
iot-control-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.77 s
iot-control-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }6.03 s
banking-ios/clean/grounding/profile3éxitoObjetivo alcanzado9.36 s
banking-ios/clean/grounding/transaction-search3éxitoObjetivo alcanzado6.13 s
banking-ios/clean/grounding/hide-balance3éxitoObjetivo alcanzado8.05 s
banking-ios/clean/grounding/transfer3éxitoObjetivo alcanzado6.87 s
banking-ios/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without cropping, clipping, overflow, overlap, or obscured controls." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects; all elements render within bounds without cropping, clipping, overflow, overlap, or obscured controls." }9.48 s
banking-ios/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 280, "height": 50 }, "description": "Page title 'Banking' is cut off at the left viewport edge, showing only 'anking'." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 0, "y": 60, "width": 280, "height": 50 }, "description": "Page title 'Banking' is cut off at the left viewport edge, showing only 'anking'." }9.28 s
banking-ios/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }4.25 s
banking-ios/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }4.52 s
flight-pass-ios/clean/grounding/boarding-pass3éxitoObjetivo alcanzado7.59 s
flight-pass-ios/clean/grounding/check-in3éxitoObjetivo alcanzado9.32 s
flight-pass-ios/clean/grounding/trips-tab3éxitoObjetivo alcanzado6.66 s
flight-pass-ios/clean/grounding/profile-tab3éxitoObjetivo alcanzado8.68 s
flight-pass-ios/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all text fits within containers, controls are unobscured, and elements do not overlap or clip." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects: all text fits within containers, controls are unobscured, and elements do not overlap or clip." }8.34 s
flight-pass-ios/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 45, "y": 850, "width": 910, "height": 70 }, "description": "A red 'Flight update' toast banner overlaps and obscures a blue action button ('Check in') near the bottom of the screen." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 45, "y": 850, "width": 910, "height": 70 }, "description": "A red 'Flight update' toast banner overlaps and obscures a blue action button ('Check in') near the bottom of the screen." }14.65 s
flight-pass-ios/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.98 s
flight-pass-ios/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }4.74 s
smart-home-android/clean/grounding/more3éxitoObjetivo alcanzado8.92 s
smart-home-android/clean/grounding/room-search3éxitoObjetivo alcanzado6.79 s
smart-home-android/clean/grounding/thermostat-card3éxitoObjetivo alcanzado7.32 s
smart-home-android/clean/grounding/add-device3éxitoObjetivo alcanzado6.50 s
smart-home-android/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "No visible layout defects detected." }8.80 s
smart-home-android/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 355, "y": 508, "width": 645, "height": 70 }, "description": "The 'Kitchen lights' quick-view card is offset to the right and its right edge is clipped by the viewport, unlike the other aligned cards." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 355, "y": 508, "width": 645, "height": 70 }, "description": "The 'Kitchen lights' quick-view card is offset to the right and its right edge is clipped by the viewport, unlike the other aligned cards." }14.15 s
smart-home-android/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.64 s
smart-home-android/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }5.56 s
Muse Spark 1.3 · 144 ejecuciones de casos

Versiones de modelo devueltas: meta/muse-spark-1.3.

Objetivos con el mismo resultado de acierto/fallo en las tres repeticiones: 23 / 24. La coincidencia también puede significar que un objetivo se falla siempre.

CasoRepeticiónResultadoDesenlaceTiempo
commerce-catalog-desktop/clean/grounding/search1éxitoObjetivo alcanzado5.13 s
commerce-catalog-desktop/clean/grounding/add-product1éxitoObjetivo alcanzado2.39 s
commerce-catalog-desktop/clean/grounding/catalog-query1éxitoObjetivo alcanzado3.01 s
commerce-catalog-desktop/clean/grounding/stock-filter1éxitoObjetivo alcanzado6.57 s
commerce-catalog-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defects observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defects observed.", "has_issue": false, "region": null }4.09 s
commerce-catalog-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Product heading clipped on left edge showing 'IANAGEMENT' and 'ducts' instead of full title.", "has_issue": true, "region": { "height": 65, "width": 68, "x": 191, "y": 126 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Product heading clipped on left edge showing 'IANAGEMENT' and 'ducts' instead of full title.", "has_issue": true, "region": { "height": 65, "width": 68, "x": 191, "y": 126 } }7.80 s
commerce-catalog-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.66 s
commerce-catalog-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.79 s
analytics-dashboard-desktop/clean/grounding/audience-tab1éxitoObjetivo alcanzado4.96 s
analytics-dashboard-desktop/clean/grounding/revenue-card1éxitoObjetivo alcanzado5.08 s
analytics-dashboard-desktop/clean/grounding/search-field1éxitoObjetivo alcanzado3.09 s
analytics-dashboard-desktop/clean/grounding/compare-toggle1éxitoObjetivo alcanzado4.94 s
analytics-dashboard-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null }4.07 s
analytics-dashboard-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card title truncated to 'Monthly revenue across all sales chann', cutting off end of word.", "has_issue": true, "region": { "height": 30, "width": 330, "x": 200, "y": 248 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card title truncated to 'Monthly revenue across all sales chann', cutting off end of word.", "has_issue": true, "region": { "height": 30, "width": 330, "x": 200, "y": 248 } }8.22 s
analytics-dashboard-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.36 s
analytics-dashboard-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.71 s
iot-control-desktop/clean/grounding/devices-nav1éxitoObjetivo alcanzado3.78 s
iot-control-desktop/clean/grounding/restart-device1éxitoObjetivo alcanzado3.54 s
iot-control-desktop/clean/grounding/auto-mode1éxitoObjetivo alcanzado6.48 s
iot-control-desktop/clean/grounding/pump-card1éxitoObjetivo alcanzado5.57 s
iot-control-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null }8.45 s
iot-control-desktop/defect/layout-defect1éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null }5.07 s
iot-control-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.95 s
iot-control-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.71 s
banking-ios/clean/grounding/profile1éxitoObjetivo alcanzado6.58 s
banking-ios/clean/grounding/transaction-search1éxitoObjetivo alcanzado3.56 s
banking-ios/clean/grounding/hide-balance1éxitoObjetivo fallado8.77 s
banking-ios/clean/grounding/transfer1éxitoObjetivo alcanzado3.70 s
banking-ios/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null }4.67 s
banking-ios/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "Header title 'banking' is cropped at left viewport edge with first letter partially cut off.", "has_issue": true, "region": { "height": 60, "width": 260, "x": 35, "y": 58 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "Header title 'banking' is cropped at left viewport edge with first letter partially cut off.", "has_issue": true, "region": { "height": 60, "width": 260, "x": 35, "y": 58 } }5.79 s
banking-ios/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.00 s
banking-ios/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.72 s
flight-pass-ios/clean/grounding/boarding-pass1éxitoObjetivo fallado4.66 s
flight-pass-ios/clean/grounding/check-in1éxitoObjetivo alcanzado3.46 s
flight-pass-ios/clean/grounding/trips-tab1éxitoObjetivo alcanzado5.15 s
flight-pass-ios/clean/grounding/profile-tab1éxitoObjetivo alcanzado4.18 s
flight-pass-ios/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null }4.41 s
flight-pass-ios/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red Flight update banner overlaps blue banner beneath it, obscuring its text.", "has_issue": true, "region": { "height": 62, "width": 912, "x": 43, "y": 863 } } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red Flight update banner overlaps blue banner beneath it, obscuring its text.", "has_issue": true, "region": { "height": 62, "width": 912, "x": 43, "y": 863 } }9.53 s
flight-pass-ios/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.16 s
flight-pass-ios/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.53 s
smart-home-android/clean/grounding/more1éxitoObjetivo alcanzado4.07 s
smart-home-android/clean/grounding/room-search1éxitoObjetivo alcanzado5.98 s
smart-home-android/clean/grounding/thermostat-card1éxitoObjetivo alcanzado4.82 s
smart-home-android/clean/grounding/add-device1éxitoObjetivo alcanzado3.71 s
smart-home-android/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null }4.54 s
smart-home-android/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": "viewport_cropping", "description": "Kitchen lights Quick view card is offset right and clipped by viewport edge.", "has_issue": true, "region": { "height": 116, "width": 845, "x": 155, "y": 534 } } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": "viewport_cropping", "description": "Kitchen lights Quick view card is offset right and clipped by viewport edge.", "has_issue": true, "region": { "height": 116, "width": 845, "x": 155, "y": 534 } }8.58 s
smart-home-android/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.20 s
smart-home-android/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }3.74 s
commerce-catalog-desktop/clean/grounding/search2éxitoObjetivo alcanzado4.15 s
commerce-catalog-desktop/clean/grounding/add-product2éxitoObjetivo alcanzado3.72 s
commerce-catalog-desktop/clean/grounding/catalog-query2éxitoObjetivo alcanzado2.27 s
commerce-catalog-desktop/clean/grounding/stock-filter2éxitoObjetivo alcanzado5.52 s
commerce-catalog-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed.", "has_issue": false, "region": null }4.06 s
commerce-catalog-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Main heading clipped on left showing 'ducts' and 'IANAGEMENT' instead of full 'Products' title.", "has_issue": true, "region": { "height": 68, "width": 69, "x": 191, "y": 125 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Main heading clipped on left showing 'ducts' and 'IANAGEMENT' instead of full 'Products' title.", "has_issue": true, "region": { "height": 68, "width": 69, "x": 191, "y": 125 } }10.11 s
commerce-catalog-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.76 s
commerce-catalog-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }3.51 s
analytics-dashboard-desktop/clean/grounding/audience-tab2éxitoObjetivo alcanzado4.69 s
analytics-dashboard-desktop/clean/grounding/revenue-card2éxitoObjetivo alcanzado3.86 s
analytics-dashboard-desktop/clean/grounding/search-field2éxitoObjetivo alcanzado4.04 s
analytics-dashboard-desktop/clean/grounding/compare-toggle2éxitoObjetivo alcanzado7.95 s
analytics-dashboard-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null }4.03 s
analytics-dashboard-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card heading truncated at container edge, reading 'chann' instead of full text.", "has_issue": true, "region": { "height": 30, "width": 328, "x": 200, "y": 249 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card heading truncated at container edge, reading 'chann' instead of full text.", "has_issue": true, "region": { "height": 30, "width": 328, "x": 200, "y": 249 } }9.23 s
analytics-dashboard-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.09 s
analytics-dashboard-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.88 s
iot-control-desktop/clean/grounding/devices-nav2éxitoObjetivo alcanzado4.10 s
iot-control-desktop/clean/grounding/restart-device2éxitoObjetivo alcanzado3.19 s
iot-control-desktop/clean/grounding/auto-mode2éxitoObjetivo alcanzado10.00 s
iot-control-desktop/clean/grounding/pump-card2éxitoObjetivo alcanzado2.78 s
iot-control-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect detected.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect detected.", "has_issue": false, "region": null }5.68 s
iot-control-desktop/defect/layout-defect2éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No visible layout defects detected.", "has_issue": false, "region": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No visible layout defects detected.", "has_issue": false, "region": null }8.11 s
iot-control-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.31 s
iot-control-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.58 s
banking-ios/clean/grounding/profile2éxitoObjetivo alcanzado4.57 s
banking-ios/clean/grounding/transaction-search2éxitoObjetivo alcanzado4.25 s
banking-ios/clean/grounding/hide-balance2éxitoObjetivo alcanzado8.61 s
banking-ios/clean/grounding/transfer2éxitoObjetivo alcanzado3.44 s
banking-ios/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect observed", "has_issue": false, "region": null }3.40 s
banking-ios/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "The 'Banking' header text is cropped at the left viewport edge, cutting off the first letter.", "has_issue": true, "region": { "height": 55, "width": 250, "x": 30, "y": 80 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "The 'Banking' header text is cropped at the left viewport edge, cutting off the first letter.", "has_issue": true, "region": { "height": 55, "width": 250, "x": 30, "y": 80 } }5.50 s
banking-ios/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }8.35 s
banking-ios/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.99 s
flight-pass-ios/clean/grounding/boarding-pass2éxitoObjetivo fallado4.49 s
flight-pass-ios/clean/grounding/check-in2éxitoObjetivo alcanzado4.15 s
flight-pass-ios/clean/grounding/trips-tab2éxitoObjetivo alcanzado6.30 s
flight-pass-ios/clean/grounding/profile-tab2éxitoObjetivo alcanzado5.09 s
flight-pass-ios/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null }4.56 s
flight-pass-ios/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red Flight update banner overlaps blue banner behind it, obscuring its text.", "has_issue": true, "region": { "height": 55, "width": 910, "x": 45, "y": 875 } } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red Flight update banner overlaps blue banner behind it, obscuring its text.", "has_issue": true, "region": { "height": 55, "width": 910, "x": 45, "y": 875 } }10.19 s
flight-pass-ios/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.61 s
flight-pass-ios/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.10 s
smart-home-android/clean/grounding/more2éxitoObjetivo alcanzado6.58 s
smart-home-android/clean/grounding/room-search2éxitoObjetivo alcanzado4.87 s
smart-home-android/clean/grounding/thermostat-card2éxitoObjetivo alcanzado4.45 s
smart-home-android/clean/grounding/add-device2éxitoObjetivo alcanzado4.54 s
smart-home-android/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect.", "has_issue": false, "region": null }4.85 s
smart-home-android/defect/layout-defect2éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": null, "description": "No layout defect detected", "has_issue": false, "region": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": null, "description": "No layout defect detected", "has_issue": false, "region": null }9.62 s
smart-home-android/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }3.87 s
smart-home-android/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.64 s
commerce-catalog-desktop/clean/grounding/search3éxitoObjetivo alcanzado4.92 s
commerce-catalog-desktop/clean/grounding/add-product3éxitoObjetivo alcanzado3.59 s
commerce-catalog-desktop/clean/grounding/catalog-query3éxitoObjetivo alcanzado3.02 s
commerce-catalog-desktop/clean/grounding/stock-filter3éxitoObjetivo alcanzado5.15 s
commerce-catalog-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect.", "has_issue": false, "region": null }4.52 s
commerce-catalog-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Page heading clipped on left edge showing 'ducts' instead of 'Products' with kicker text cut off.", "has_issue": true, "region": { "height": 68, "width": 69, "x": 190, "y": 124 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "category": "container_clipping", "description": "Page heading clipped on left edge showing 'ducts' instead of 'Products' with kicker text cut off.", "has_issue": true, "region": { "height": 68, "width": 69, "x": 190, "y": 124 } }5.27 s
commerce-catalog-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.90 s
commerce-catalog-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }4.55 s
analytics-dashboard-desktop/clean/grounding/audience-tab3éxitoObjetivo alcanzado5.64 s
analytics-dashboard-desktop/clean/grounding/revenue-card3éxitoObjetivo alcanzado3.06 s
analytics-dashboard-desktop/clean/grounding/search-field3éxitoObjetivo alcanzado3.71 s
analytics-dashboard-desktop/clean/grounding/compare-toggle3éxitoObjetivo alcanzado5.52 s
analytics-dashboard-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect detected.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No visible layout defect detected.", "has_issue": false, "region": null }3.62 s
analytics-dashboard-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card heading truncated to 'Monthly revenue across all sales chann', clipped at container edge.", "has_issue": true, "region": { "height": 30, "width": 330, "x": 200, "y": 250 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "category": "text_overflow", "description": "Card heading truncated to 'Monthly revenue across all sales chann', clipped at container edge.", "has_issue": true, "region": { "height": 30, "width": 330, "x": 200, "y": 250 } }8.87 s
analytics-dashboard-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.07 s
analytics-dashboard-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.90 s
iot-control-desktop/clean/grounding/devices-nav3éxitoObjetivo alcanzado3.49 s
iot-control-desktop/clean/grounding/restart-device3éxitoObjetivo alcanzado4.07 s
iot-control-desktop/clean/grounding/auto-mode3éxitoObjetivo alcanzado5.22 s
iot-control-desktop/clean/grounding/pump-card3éxitoObjetivo alcanzado5.83 s
iot-control-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect detected.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect detected.", "has_issue": false, "region": null }4.75 s
iot-control-desktop/defect/layout-defect3éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No layout defect detected", "has_issue": false, "region": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "category": null, "description": "No layout defect detected", "has_issue": false, "region": null }7.46 s
iot-control-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }4.12 s
iot-control-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.87 s
banking-ios/clean/grounding/profile3éxitoObjetivo alcanzado4.00 s
banking-ios/clean/grounding/transaction-search3éxitoObjetivo alcanzado5.64 s
banking-ios/clean/grounding/hide-balance3éxitoObjetivo alcanzado6.21 s
banking-ios/clean/grounding/transfer3éxitoObjetivo alcanzado3.53 s
banking-ios/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect visible.", "has_issue": false, "region": null }5.26 s
banking-ios/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "Header title Banking and top brand text truncated at left viewport edge with B cropped.", "has_issue": true, "region": { "height": 75, "width": 270, "x": 10, "y": 35 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "category": "viewport_cropping", "description": "Header title Banking and top brand text truncated at left viewport edge with B cropped.", "has_issue": true, "region": { "height": 75, "width": 270, "x": 10, "y": 35 } }7.26 s
banking-ios/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.34 s
banking-ios/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }3.49 s
flight-pass-ios/clean/grounding/boarding-pass3éxitoObjetivo fallado3.76 s
flight-pass-ios/clean/grounding/check-in3éxitoObjetivo alcanzado4.72 s
flight-pass-ios/clean/grounding/trips-tab3éxitoObjetivo alcanzado4.97 s
flight-pass-ios/clean/grounding/profile-tab3éxitoObjetivo alcanzado5.65 s
flight-pass-ios/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect detected.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defect detected.", "has_issue": false, "region": null }5.01 s
flight-pass-ios/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red flight update banner overlaps blue banner behind it, obscuring its text", "has_issue": true, "region": { "height": 75, "width": 920, "x": 40, "y": 840 } } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "category": "element_overlap", "description": "Red flight update banner overlaps blue banner behind it, obscuring its text", "has_issue": true, "region": { "height": 75, "width": 920, "x": 40, "y": 840 } }5.54 s
flight-pass-ios/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.05 s
flight-pass-ios/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.99 s
smart-home-android/clean/grounding/more3éxitoObjetivo alcanzado10.73 s
smart-home-android/clean/grounding/room-search3éxitoObjetivo alcanzado4.38 s
smart-home-android/clean/grounding/thermostat-card3éxitoObjetivo alcanzado4.20 s
smart-home-android/clean/grounding/add-device3éxitoObjetivo alcanzado3.82 s
smart-home-android/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defects observed.", "has_issue": false, "region": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "category": null, "description": "No layout defects observed.", "has_issue": false, "region": null }4.26 s
smart-home-android/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": "broken_responsive_reflow", "description": "The Kitchen lights card is indented and narrower than other cards, breaking list alignment.", "has_issue": true, "region": { "height": 90, "width": 675, "x": 280, "y": 530 } } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "category": "broken_responsive_reflow", "description": "The Kitchen lights card is indented and narrower than other cards, breaking list alignment.", "has_issue": true, "region": { "height": 90, "width": 675, "x": 280, "y": 530 } }11.76 s
smart-home-android/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.94 s
smart-home-android/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.33 s
GPT-5.6 Luna · 144 ejecuciones de casos

Versiones de modelo devueltas: openai/gpt-5.6-luna.

Objetivos con el mismo resultado de acierto/fallo en las tres repeticiones: 23 / 24. La coincidencia también puede significar que un objetivo se falla siempre.

CasoRepeticiónResultadoDesenlaceTiempo
commerce-catalog-desktop/clean/grounding/search1éxitoObjetivo alcanzado2.56 s
commerce-catalog-desktop/clean/grounding/add-product1éxitoObjetivo alcanzado2.00 s
commerce-catalog-desktop/clean/grounding/catalog-query1éxitoObjetivo alcanzado2.45 s
commerce-catalog-desktop/clean/grounding/stock-filter1éxitoObjetivo alcanzado2.89 s
commerce-catalog-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.03 s
commerce-catalog-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 127, "width": 70, "height": 62 }, "description": "The left portions of the page heading text are visibly clipped, showing “IAGEMENT” and “ducts” instead of the full labels." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 127, "width": 70, "height": 62 }, "description": "The left portions of the page heading text are visibly clipped, showing “IAGEMENT” and “ducts” instead of the full labels." }4.66 s
commerce-catalog-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.85 s
commerce-catalog-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.73 s
analytics-dashboard-desktop/clean/grounding/audience-tab1éxitoObjetivo alcanzado5.28 s
analytics-dashboard-desktop/clean/grounding/revenue-card1éxitoObjetivo alcanzado2.12 s
analytics-dashboard-desktop/clean/grounding/search-field1éxitoObjetivo alcanzado1.87 s
analytics-dashboard-desktop/clean/grounding/compare-toggle1éxitoObjetivo alcanzado3.44 s
analytics-dashboard-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.97 s
analytics-dashboard-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 250, "width": 326, "height": 34 }, "description": "The first summary card's heading is visibly truncated at the card's right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 250, "width": 326, "height": 34 }, "description": "The first summary card's heading is visibly truncated at the card's right edge." }3.08 s
analytics-dashboard-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.31 s
analytics-dashboard-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.29 s
iot-control-desktop/clean/grounding/devices-nav1éxitoObjetivo alcanzado2.47 s
iot-control-desktop/clean/grounding/restart-device1éxitoObjetivo alcanzado2.03 s
iot-control-desktop/clean/grounding/auto-mode1éxitoObjetivo alcanzado4.02 s
iot-control-desktop/clean/grounding/pump-card1éxitoObjetivo alcanzado2.64 s
iot-control-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.91 s
iot-control-desktop/defect/layout-defect1éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }3.79 s
iot-control-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.55 s
iot-control-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.29 s
banking-ios/clean/grounding/profile1éxitoObjetivo alcanzado2.82 s
banking-ios/clean/grounding/transaction-search1éxitoObjetivo fallado1.87 s
banking-ios/clean/grounding/hide-balance1éxitoObjetivo alcanzado3.78 s
banking-ios/clean/grounding/transfer1éxitoObjetivo alcanzado1.69 s
banking-ios/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.95 s
banking-ios/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 44, "y": 71, "width": 244, "height": 33 }, "description": "The left edge of the large “Banking” heading is visibly cropped, leaving the initial character partially cut off." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 44, "y": 71, "width": 244, "height": 33 }, "description": "The left edge of the large “Banking” heading is visibly cropped, leaving the initial character partially cut off." }6.58 s
banking-ios/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.64 s
banking-ios/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.33 s
flight-pass-ios/clean/grounding/boarding-pass1éxitoObjetivo alcanzado2.10 s
flight-pass-ios/clean/grounding/check-in1éxitoObjetivo alcanzado2.62 s
flight-pass-ios/clean/grounding/trips-tab1éxitoObjetivo alcanzado4.46 s
flight-pass-ios/clean/grounding/profile-tab1éxitoObjetivo alcanzado5.20 s
flight-pass-ios/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.61 s
flight-pass-ios/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "A red flight update banner visibly covers the lower portion of a blue actionable button near the bottom of the screen." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 874, "width": 908, "height": 40 }, "description": "A red flight update banner visibly covers the lower portion of a blue actionable button near the bottom of the screen." }3.27 s
flight-pass-ios/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.40 s
flight-pass-ios/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.14 s
smart-home-android/clean/grounding/more1éxitoObjetivo alcanzado2.51 s
smart-home-android/clean/grounding/room-search1éxitoObjetivo alcanzado1.95 s
smart-home-android/clean/grounding/thermostat-card1éxitoObjetivo alcanzado2.74 s
smart-home-android/clean/grounding/add-device1éxitoObjetivo alcanzado2.44 s
smart-home-android/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.88 s
smart-home-android/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 510, "width": 600, "height": 69 }, "description": "The second Quick view card is visibly offset to the right and does not align with the other cards in the list." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 510, "width": 600, "height": 69 }, "description": "The second Quick view card is visibly offset to the right and does not align with the other cards in the list." }5.45 s
smart-home-android/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.57 s
smart-home-android/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.57 s
commerce-catalog-desktop/clean/grounding/search2éxitoObjetivo alcanzado2.62 s
commerce-catalog-desktop/clean/grounding/add-product2éxitoObjetivo alcanzado2.64 s
commerce-catalog-desktop/clean/grounding/catalog-query2éxitoObjetivo alcanzado2.37 s
commerce-catalog-desktop/clean/grounding/stock-filter2éxitoObjetivo alcanzado3.21 s
commerce-catalog-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.91 s
commerce-catalog-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 192, "y": 151, "width": 72, "height": 39 }, "description": "The page title is visibly clipped on the left, displaying only “ducts” instead of the full title." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 192, "y": 151, "width": 72, "height": 39 }, "description": "The page title is visibly clipped on the left, displaying only “ducts” instead of the full title." }4.33 s
commerce-catalog-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.17 s
commerce-catalog-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.70 s
analytics-dashboard-desktop/clean/grounding/audience-tab2éxitoObjetivo alcanzado4.50 s
analytics-dashboard-desktop/clean/grounding/revenue-card2éxitoObjetivo alcanzado2.67 s
analytics-dashboard-desktop/clean/grounding/search-field2éxitoObjetivo alcanzado2.36 s
analytics-dashboard-desktop/clean/grounding/compare-toggle2éxitoObjetivo alcanzado3.04 s
analytics-dashboard-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.84 s
analytics-dashboard-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 252, "width": 325, "height": 28 }, "description": "The first summary card's headline is visibly truncated at the right edge, ending mid-word." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 202, "y": 252, "width": 325, "height": 28 }, "description": "The first summary card's headline is visibly truncated at the right edge, ending mid-word." }2.88 s
analytics-dashboard-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.57 s
analytics-dashboard-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.66 s
iot-control-desktop/clean/grounding/devices-nav2éxitoObjetivo alcanzado3.05 s
iot-control-desktop/clean/grounding/restart-device2éxitoObjetivo alcanzado2.25 s
iot-control-desktop/clean/grounding/auto-mode2éxitoObjetivo alcanzado2.80 s
iot-control-desktop/clean/grounding/pump-card2éxitoObjetivo alcanzado3.40 s
iot-control-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.31 s
iot-control-desktop/defect/layout-defect2éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.85 s
iot-control-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }2.08 s
iot-control-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }2.07 s
banking-ios/clean/grounding/profile2éxitoObjetivo alcanzado3.15 s
banking-ios/clean/grounding/transaction-search2éxitoObjetivo alcanzado2.17 s
banking-ios/clean/grounding/hide-balance2éxitoObjetivo alcanzado3.60 s
banking-ios/clean/grounding/transfer2éxitoObjetivo alcanzado2.21 s
banking-ios/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.69 s
banking-ios/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 45, "y": 70, "width": 245, "height": 32 }, "description": "The left edge of the “Banking” header text is visibly clipped, with the initial letter cut off." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 45, "y": 70, "width": 245, "height": 32 }, "description": "The left edge of the “Banking” header text is visibly clipped, with the initial letter cut off." }4.84 s
banking-ios/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.59 s
banking-ios/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.55 s
flight-pass-ios/clean/grounding/boarding-pass2éxitoObjetivo alcanzado3.22 s
flight-pass-ios/clean/grounding/check-in2éxitoObjetivo alcanzado2.06 s
flight-pass-ios/clean/grounding/trips-tab2éxitoObjetivo alcanzado7.16 s
flight-pass-ios/clean/grounding/profile-tab2éxitoObjetivo alcanzado3.82 s
flight-pass-ios/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.55 s
flight-pass-ios/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 64 }, "description": "A red flight-update banner visibly overlaps and obscures the blue actionable button near the bottom of the screen." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 850, "width": 908, "height": 64 }, "description": "A red flight-update banner visibly overlaps and obscures the blue actionable button near the bottom of the screen." }3.37 s
flight-pass-ios/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.73 s
flight-pass-ios/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.64 s
smart-home-android/clean/grounding/more2éxitoObjetivo alcanzado2.67 s
smart-home-android/clean/grounding/room-search2éxitoObjetivo alcanzado2.30 s
smart-home-android/clean/grounding/thermostat-card2éxitoObjetivo alcanzado2.26 s
smart-home-android/clean/grounding/add-device2éxitoObjetivo alcanzado1.93 s
smart-home-android/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.57 s
smart-home-android/defect/layout-defect2éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.87 s
smart-home-android/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.17 s
smart-home-android/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.36 s
commerce-catalog-desktop/clean/grounding/search3éxitoObjetivo alcanzado2.61 s
commerce-catalog-desktop/clean/grounding/add-product3éxitoObjetivo alcanzado2.88 s
commerce-catalog-desktop/clean/grounding/catalog-query3éxitoObjetivo alcanzado4.31 s
commerce-catalog-desktop/clean/grounding/stock-filter3éxitoObjetivo alcanzado3.36 s
commerce-catalog-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.36 s
commerce-catalog-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 76, "height": 64 }, "description": "The page title is visibly clipped on the left, displaying “ducts” instead of the full “Products”." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 192, "y": 126, "width": 76, "height": 64 }, "description": "The page title is visibly clipped on the left, displaying “ducts” instead of the full “Products”." }3.66 s
commerce-catalog-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.69 s
commerce-catalog-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.80 s
analytics-dashboard-desktop/clean/grounding/audience-tab3éxitoObjetivo alcanzado3.61 s
analytics-dashboard-desktop/clean/grounding/revenue-card3éxitoObjetivo alcanzado2.33 s
analytics-dashboard-desktop/clean/grounding/search-field3éxitoObjetivo alcanzado2.09 s
analytics-dashboard-desktop/clean/grounding/compare-toggle3éxitoObjetivo alcanzado3.14 s
analytics-dashboard-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.33 s
analytics-dashboard-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 253, "width": 326, "height": 28 }, "description": "The first summary card's heading is visibly cut off at the right edge." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 201, "y": 253, "width": 326, "height": 28 }, "description": "The first summary card's heading is visibly cut off at the right edge." }3.52 s
analytics-dashboard-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.61 s
analytics-dashboard-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.64 s
iot-control-desktop/clean/grounding/devices-nav3éxitoObjetivo alcanzado2.44 s
iot-control-desktop/clean/grounding/restart-device3éxitoObjetivo alcanzado3.00 s
iot-control-desktop/clean/grounding/auto-mode3éxitoObjetivo alcanzado3.80 s
iot-control-desktop/clean/grounding/pump-card3éxitoObjetivo alcanzado3.33 s
iot-control-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.57 s
iot-control-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 463, "y": 558, "width": 222, "height": 66 }, "description": "A maintenance notification overlays the device table content." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 463, "y": 558, "width": 222, "height": 66 }, "description": "A maintenance notification overlays the device table content." }3.65 s
iot-control-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.82 s
iot-control-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.48 s
banking-ios/clean/grounding/profile3éxitoObjetivo alcanzado2.94 s
banking-ios/clean/grounding/transaction-search3éxitoObjetivo alcanzado2.50 s
banking-ios/clean/grounding/hide-balance3éxitoObjetivo alcanzado4.97 s
banking-ios/clean/grounding/transfer3éxitoObjetivo alcanzado2.55 s
banking-ios/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }2.96 s
banking-ios/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 44, "y": 72, "width": 244, "height": 29 }, "description": "The left edge of the Banking heading is visibly clipped, cutting off the first character." } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "container_clipping", "region": { "x": 44, "y": 72, "width": 244, "height": 29 }, "description": "The left edge of the Banking heading is visibly clipped, cutting off the first character." }3.82 s
banking-ios/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.26 s
banking-ios/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.30 s
flight-pass-ios/clean/grounding/boarding-pass3éxitoObjetivo alcanzado2.52 s
flight-pass-ios/clean/grounding/check-in3éxitoObjetivo alcanzado2.24 s
flight-pass-ios/clean/grounding/trips-tab3éxitoObjetivo alcanzado4.28 s
flight-pass-ios/clean/grounding/profile-tab3éxitoObjetivo alcanzado3.22 s
flight-pass-ios/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.70 s
flight-pass-ios/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 63 }, "description": "A red flight update banner obscures the lower portion of a blue actionable button near the bottom of the screen." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46, "y": 851, "width": 908, "height": 63 }, "description": "A red flight update banner obscures the lower portion of a blue actionable button near the bottom of the screen." }3.14 s
flight-pass-ios/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.40 s
flight-pass-ios/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.32 s
smart-home-android/clean/grounding/more3éxitoObjetivo alcanzado2.35 s
smart-home-android/clean/grounding/room-search3éxitoObjetivo alcanzado2.65 s
smart-home-android/clean/grounding/thermostat-card3éxitoObjetivo alcanzado2.64 s
smart-home-android/clean/grounding/add-device3éxitoObjetivo alcanzado1.92 s
smart-home-android/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }1.88 s
smart-home-android/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 510, "width": 600, "height": 69 }, "description": "The second Quick view card is misaligned and compressed to the right compared with the other full-width cards." } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 354, "y": 510, "width": 600, "height": 69 }, "description": "The second Quick view card is misaligned and compressed to the right compared with the other full-width cards." }4.82 s
smart-home-android/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.19 s
smart-home-android/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }1.62 s
Jev 1.14 · 144 ejecuciones de casos

Versiones de modelo devueltas: typesafe/jev-1.14-20261014.

Objetivos con el mismo resultado de acierto/fallo en las tres repeticiones: 24 / 24. La coincidencia también puede significar que un objetivo se falla siempre.

CasoRepeticiónResultadoDesenlaceTiempo
commerce-catalog-desktop/clean/grounding/search1éxitoObjetivo alcanzado0.43 s
commerce-catalog-desktop/clean/grounding/add-product1éxitoObjetivo alcanzado0.41 s
commerce-catalog-desktop/clean/grounding/catalog-query1éxitoObjetivo alcanzado0.40 s
commerce-catalog-desktop/clean/grounding/stock-filter1éxitoObjetivo alcanzado0.38 s
commerce-catalog-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.47 s
commerce-catalog-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 190, "y": 140, "width": 60, "height": 69.99999999999999 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 190, "y": 140, "width": 60, "height": 69.99999999999999 }, "description": "" }0.39 s
commerce-catalog-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.35 s
commerce-catalog-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.44 s
analytics-dashboard-desktop/clean/grounding/audience-tab1éxitoObjetivo alcanzado0.53 s
analytics-dashboard-desktop/clean/grounding/revenue-card1éxitoObjetivo alcanzado0.37 s
analytics-dashboard-desktop/clean/grounding/search-field1éxitoObjetivo alcanzado0.38 s
analytics-dashboard-desktop/clean/grounding/compare-toggle1éxitoObjetivo alcanzado0.34 s
analytics-dashboard-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.48 s
analytics-dashboard-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 360.00000000000006, "height": 110.00000000000001 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 360.00000000000006, "height": 110.00000000000001 }, "description": "" }0.36 s
analytics-dashboard-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.33 s
analytics-dashboard-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.39 s
iot-control-desktop/clean/grounding/devices-nav1éxitoObjetivo alcanzado0.34 s
iot-control-desktop/clean/grounding/restart-device1éxitoObjetivo alcanzado0.32 s
iot-control-desktop/clean/grounding/auto-mode1éxitoObjetivo alcanzado0.41 s
iot-control-desktop/clean/grounding/pump-card1éxitoObjetivo alcanzado0.37 s
iot-control-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.48 s
iot-control-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 450, "y": 550, "width": 239.99999999999994, "height": 89.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 450, "y": 550, "width": 239.99999999999994, "height": 89.99999999999997 }, "description": "" }0.38 s
iot-control-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.35 s
iot-control-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.46 s
banking-ios/clean/grounding/profile1éxitoObjetivo alcanzado0.39 s
banking-ios/clean/grounding/transaction-search1éxitoObjetivo alcanzado0.48 s
banking-ios/clean/grounding/hide-balance1éxitoObjetivo alcanzado0.41 s
banking-ios/clean/grounding/transfer1éxitoObjetivo alcanzado0.43 s
banking-ios/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.37 s
banking-ios/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" }0.30 s
banking-ios/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.37 s
banking-ios/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.38 s
flight-pass-ios/clean/grounding/boarding-pass1éxitoObjetivo alcanzado0.31 s
flight-pass-ios/clean/grounding/check-in1éxitoObjetivo alcanzado0.48 s
flight-pass-ios/clean/grounding/trips-tab1éxitoObjetivo alcanzado0.34 s
flight-pass-ios/clean/grounding/profile-tab1éxitoObjetivo alcanzado0.32 s
flight-pass-ios/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.30 s
flight-pass-ios/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 70.00000000000006 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 70.00000000000006 }, "description": "" }0.37 s
flight-pass-ios/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.28 s
flight-pass-ios/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.38 s
smart-home-android/clean/grounding/more1éxitoObjetivo alcanzado0.36 s
smart-home-android/clean/grounding/room-search1éxitoObjetivo alcanzado0.34 s
smart-home-android/clean/grounding/thermostat-card1éxitoObjetivo alcanzado0.37 s
smart-home-android/clean/grounding/add-device1éxitoObjetivo alcanzado0.33 s
smart-home-android/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.43 s
smart-home-android/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 470, "width": 590, "height": 99.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 470, "width": 590, "height": 99.99999999999997 }, "description": "" }0.39 s
smart-home-android/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.38 s
smart-home-android/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.41 s
commerce-catalog-desktop/clean/grounding/search2éxitoObjetivo alcanzado0.44 s
commerce-catalog-desktop/clean/grounding/add-product2éxitoObjetivo alcanzado0.42 s
commerce-catalog-desktop/clean/grounding/catalog-query2éxitoObjetivo alcanzado0.38 s
commerce-catalog-desktop/clean/grounding/stock-filter2éxitoObjetivo alcanzado0.42 s
commerce-catalog-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.38 s
commerce-catalog-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 180, "y": 140, "width": 70, "height": 69.99999999999999 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 180, "y": 140, "width": 70, "height": 69.99999999999999 }, "description": "" }0.39 s
commerce-catalog-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.38 s
commerce-catalog-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.54 s
analytics-dashboard-desktop/clean/grounding/audience-tab2éxitoObjetivo alcanzado0.41 s
analytics-dashboard-desktop/clean/grounding/revenue-card2éxitoObjetivo alcanzado0.66 s
analytics-dashboard-desktop/clean/grounding/search-field2éxitoObjetivo alcanzado0.48 s
analytics-dashboard-desktop/clean/grounding/compare-toggle2éxitoObjetivo alcanzado0.37 s
analytics-dashboard-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.39 s
analytics-dashboard-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 360.00000000000006, "height": 100 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 360.00000000000006, "height": 100 }, "description": "" }0.39 s
analytics-dashboard-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.36 s
analytics-dashboard-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.87 s
iot-control-desktop/clean/grounding/devices-nav2éxitoObjetivo alcanzado0.46 s
iot-control-desktop/clean/grounding/restart-device2éxitoObjetivo alcanzado0.66 s
iot-control-desktop/clean/grounding/auto-mode2éxitoObjetivo alcanzado0.39 s
iot-control-desktop/clean/grounding/pump-card2éxitoObjetivo alcanzado0.35 s
iot-control-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.33 s
iot-control-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 460, "y": 540, "width": 239.99999999999994, "height": 99.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 460, "y": 540, "width": 239.99999999999994, "height": 99.99999999999997 }, "description": "" }0.37 s
iot-control-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.33 s
iot-control-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.38 s
banking-ios/clean/grounding/profile2éxitoObjetivo alcanzado0.33 s
banking-ios/clean/grounding/transaction-search2éxitoObjetivo alcanzado0.30 s
banking-ios/clean/grounding/hide-balance2éxitoObjetivo alcanzado0.33 s
banking-ios/clean/grounding/transfer2éxitoObjetivo alcanzado0.41 s
banking-ios/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.30 s
banking-ios/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" }0.44 s
banking-ios/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.31 s
banking-ios/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.29 s
flight-pass-ios/clean/grounding/boarding-pass2éxitoObjetivo alcanzado0.30 s
flight-pass-ios/clean/grounding/check-in2éxitoObjetivo alcanzado0.29 s
flight-pass-ios/clean/grounding/trips-tab2éxitoObjetivo alcanzado0.27 s
flight-pass-ios/clean/grounding/profile-tab2éxitoObjetivo alcanzado0.31 s
flight-pass-ios/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.32 s
flight-pass-ios/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 70.00000000000006 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 70.00000000000006 }, "description": "" }0.31 s
flight-pass-ios/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.29 s
flight-pass-ios/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.29 s
smart-home-android/clean/grounding/more2éxitoObjetivo alcanzado0.32 s
smart-home-android/clean/grounding/room-search2éxitoObjetivo alcanzado0.30 s
smart-home-android/clean/grounding/thermostat-card2éxitoObjetivo alcanzado0.38 s
smart-home-android/clean/grounding/add-device2éxitoObjetivo alcanzado0.32 s
smart-home-android/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.29 s
smart-home-android/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 480, "width": 590, "height": 99.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 480, "width": 590, "height": 99.99999999999997 }, "description": "" }0.30 s
smart-home-android/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.32 s
smart-home-android/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.35 s
commerce-catalog-desktop/clean/grounding/search3éxitoObjetivo alcanzado0.35 s
commerce-catalog-desktop/clean/grounding/add-product3éxitoObjetivo alcanzado0.37 s
commerce-catalog-desktop/clean/grounding/catalog-query3éxitoObjetivo alcanzado0.40 s
commerce-catalog-desktop/clean/grounding/stock-filter3éxitoObjetivo alcanzado0.45 s
commerce-catalog-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.37 s
commerce-catalog-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 180, "y": 140, "width": 70, "height": 69.99999999999999 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "container_clipping", "region": { "x": 193.0556, "y": 124.4444, "width": 291.6667, "height": 74.4444 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 180, "y": 140, "width": 70, "height": 69.99999999999999 }, "description": "" }0.38 s
commerce-catalog-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.39 s
commerce-catalog-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.43 s
analytics-dashboard-desktop/clean/grounding/audience-tab3éxitoObjetivo alcanzado0.33 s
analytics-dashboard-desktop/clean/grounding/revenue-card3éxitoObjetivo alcanzado0.35 s
analytics-dashboard-desktop/clean/grounding/search-field3éxitoObjetivo alcanzado0.36 s
analytics-dashboard-desktop/clean/grounding/compare-toggle3éxitoObjetivo alcanzado0.38 s
analytics-dashboard-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.35 s
analytics-dashboard-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 350.00000000000006, "height": 110.00000000000001 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow", "region": { "x": 187.5, "y": 223.3333, "width": 340.115, "height": 144.4444 } } Returned: { "has_issue": true, "category": "text_overflow", "region": { "x": 180, "y": 230, "width": 350.00000000000006, "height": 110.00000000000001 }, "description": "" }0.36 s
analytics-dashboard-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.36 s
analytics-dashboard-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.39 s
iot-control-desktop/clean/grounding/devices-nav3éxitoObjetivo alcanzado0.33 s
iot-control-desktop/clean/grounding/restart-device3éxitoObjetivo alcanzado0.32 s
iot-control-desktop/clean/grounding/auto-mode3éxitoObjetivo alcanzado0.34 s
iot-control-desktop/clean/grounding/pump-card3éxitoObjetivo alcanzado0.35 s
iot-control-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.37 s
iot-control-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 460, "y": 550, "width": 229.99999999999991, "height": 89.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap", "region": { "x": 188.1944, "y": 515.5556, "width": 783.3333, "height": 240 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 460, "y": 550, "width": 229.99999999999991, "height": 89.99999999999997 }, "description": "" }0.37 s
iot-control-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.36 s
iot-control-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.44 s
banking-ios/clean/grounding/profile3éxitoObjetivo alcanzado0.29 s
banking-ios/clean/grounding/transaction-search3éxitoObjetivo alcanzado0.29 s
banking-ios/clean/grounding/hide-balance3éxitoObjetivo alcanzado0.29 s
banking-ios/clean/grounding/transfer3éxitoObjetivo alcanzado0.33 s
banking-ios/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.44 s
banking-ios/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 46.1538, "y": 46.2085, "width": 730.7692, "height": 73.4597 } } Returned: { "has_issue": true, "category": "viewport_cropping", "region": { "x": 50, "y": 70, "width": 240, "height": 39.99999999999999 }, "description": "" }0.30 s
banking-ios/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.28 s
banking-ios/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.44 s
flight-pass-ios/clean/grounding/boarding-pass3éxitoObjetivo alcanzado0.32 s
flight-pass-ios/clean/grounding/check-in3éxitoObjetivo alcanzado0.30 s
flight-pass-ios/clean/grounding/trips-tab3éxitoObjetivo alcanzado0.29 s
flight-pass-ios/clean/grounding/profile-tab3éxitoObjetivo alcanzado0.28 s
flight-pass-ios/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.36 s
flight-pass-ios/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 60.00000000000006 }, "description": "" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control", "region": { "x": 46.1538, "y": 850.7109, "width": 907.6923, "height": 63.981 } } Returned: { "has_issue": true, "category": "element_overlap", "region": { "x": 40, "y": 850, "width": 909.9999999999999, "height": 60.00000000000006 }, "description": "" }0.30 s
flight-pass-ios/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.29 s
flight-pass-ios/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.29 s
smart-home-android/clean/grounding/more3éxitoObjetivo alcanzado0.29 s
smart-home-android/clean/grounding/room-search3éxitoObjetivo alcanzado0.39 s
smart-home-android/clean/grounding/thermostat-card3éxitoObjetivo alcanzado0.35 s
smart-home-android/clean/grounding/add-device3éxitoObjetivo alcanzado0.35 s
smart-home-android/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null, "region": null } Returned: { "has_issue": false, "category": null, "region": null, "description": "" }0.35 s
smart-home-android/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 480, "width": 590, "height": 99.99999999999997 }, "description": "" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 46.1538, "y": 390.9953, "width": 907.6923, "height": 274.8815 } } Returned: { "has_issue": true, "category": "broken_responsive_reflow", "region": { "x": 350, "y": 480, "width": 590, "height": 99.99999999999997 }, "description": "" }0.53 s
smart-home-android/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.39 s
smart-home-android/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.34 s
GPT-6 Luna Decisions · 72 ejecuciones de casos

Versiones de modelo devueltas: openai/gpt-6-luna-decisions-20261006.

CasoRepeticiónResultadoDesenlaceTiempo
commerce-catalog-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.38 s
commerce-catalog-desktop/defect/layout-defect1éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null }0.45 s
commerce-catalog-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.32 s
commerce-catalog-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.35 s
analytics-dashboard-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.38 s
analytics-dashboard-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" }0.36 s
analytics-dashboard-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.30 s
analytics-dashboard-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.38 s
iot-control-desktop/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.32 s
iot-control-desktop/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" }0.52 s
iot-control-desktop/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.55 s
iot-control-desktop/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.26 s
banking-ios/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.24 s
banking-ios/defect/layout-defect1éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null }0.21 s
banking-ios/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.29 s
banking-ios/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.22 s
flight-pass-ios/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.53 s
flight-pass-ios/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" }0.66 s
flight-pass-ios/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.38 s
flight-pass-ios/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.19 s
smart-home-android/clean/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.33 s
smart-home-android/defect/layout-defect1éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" }0.27 s
smart-home-android/clean/element-presence/present1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.26 s
smart-home-android/clean/element-presence/absent1éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.21 s
commerce-catalog-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.35 s
commerce-catalog-desktop/defect/layout-defect2éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null }0.38 s
commerce-catalog-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.27 s
commerce-catalog-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.36 s
analytics-dashboard-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.35 s
analytics-dashboard-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" }0.32 s
analytics-dashboard-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.47 s
analytics-dashboard-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.28 s
iot-control-desktop/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.74 s
iot-control-desktop/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" }0.60 s
iot-control-desktop/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.91 s
iot-control-desktop/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.41 s
banking-ios/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.24 s
banking-ios/defect/layout-defect2éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null }0.21 s
banking-ios/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.49 s
banking-ios/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.24 s
flight-pass-ios/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.62 s
flight-pass-ios/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" }0.27 s
flight-pass-ios/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.26 s
flight-pass-ios/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.23 s
smart-home-android/clean/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.24 s
smart-home-android/defect/layout-defect2éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" }0.53 s
smart-home-android/clean/element-presence/present2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.29 s
smart-home-android/clean/element-presence/absent2éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.19 s
commerce-catalog-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.49 s
commerce-catalog-desktop/defect/layout-defect3éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "container_clipping" } Returned: { "has_issue": false, "category": null }0.25 s
commerce-catalog-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.50 s
commerce-catalog-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.48 s
analytics-dashboard-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.36 s
analytics-dashboard-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "text_overflow" } Returned: { "has_issue": true, "category": "text_overflow" }0.23 s
analytics-dashboard-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }1.32 s
analytics-dashboard-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.52 s
iot-control-desktop/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.36 s
iot-control-desktop/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "element_overlap" } Returned: { "has_issue": true, "category": "element_overlap" }0.37 s
iot-control-desktop/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.31 s
iot-control-desktop/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.28 s
banking-ios/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.25 s
banking-ios/defect/layout-defect3éxitoThe model’s presence decision differs from the expected answer. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null } · Incorrect: a defect is present, but the model reported no issue, so it did not identify the expected defect category. Expected: { "has_issue": true, "category": "viewport_cropping" } Returned: { "has_issue": false, "category": null }0.25 s
banking-ios/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.27 s
banking-ios/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.52 s
flight-pass-ios/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }0.54 s
flight-pass-ios/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" } · Incorrect: the returned defect category does not match the seeded defect category. Expected: { "has_issue": true, "category": "obscured_actionable_control" } Returned: { "has_issue": true, "category": "element_overlap" }1.22 s
flight-pass-ios/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.19 s
flight-pass-ios/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.89 s
smart-home-android/clean/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null } · Not applicable: this is a clean control screen. Defect-category accuracy is scored only on screens with a seeded defect, even if a model reports a false alarm. Expected: { "has_issue": false, "category": null } Returned: { "has_issue": false, "category": null }1.09 s
smart-home-android/defect/layout-defect3éxitoThe model’s presence decision matches the expected answer. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" } · The model identified the expected defect category. Expected: { "has_issue": true, "category": "broken_responsive_reflow" } Returned: { "has_issue": true, "category": "broken_responsive_reflow" }0.25 s
smart-home-android/clean/element-presence/present3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": true } Returned: { "found": true }0.21 s
smart-home-android/clean/element-presence/absent3éxitoThe model’s presence decision matches the expected answer. Expected: { "found": false } Returned: { "found": false }0.32 s

El piloto completado

Ejecuciones de casos registradas: 1368 / 1368. Intentos HTTP: 1368. Coste de API declarado: $3.3802.

Nueve configuraciones pueden devolver coordenadas de objetivos. GPT-6 Luna Decisions responde a preguntas de clasificación y se excluye de los resultados de localización.

Metodología

Mide lo que el modelo realmente ve

Las secciones de experimentos documentan sus propias reglas de puntuación. Esta metodología compartida abarca el contrato de entrada común, las operaciones de los proveedores, el contexto de la investigación y sus limitaciones.

EntradaSolo capturas de pantalla
Geometría0–1000
EjecucionesTres por caso
PuntuaciónMétricas independientes

Operaciones

Tiempo de respuesta, fallos y coste

Ida y vuelta HTTP y análisis local de los intentos finales conservados, sin consultas de metadatos de generación. Todos los intentos de transporte contribuyen al coste declarado. La configuración de solo clasificación ejecuta la mitad de casos.

ModeloCompletados / previstosFallos finales de solicitudesRespuestas inválidasMedianaP95Coste declaradoCoste por 1.000 solicitudes
GPT-6 Luna144 / 144002.18 s3.83 s$0.0187$0.1296
GPT-6.1 Sol144 / 144002.68 s4.88 s$0.2803$1.9468
Claude Sonnet 5.5144 / 144082.20 s3.99 s$0.5803$4.0297
Gemini 3.8 Flash144 / 144012.42 s10.96 s$0.2655$1.8437
Grok 4.7144 / 1440212.03 s28.38 s$1.1171$7.7580
Qwen3.8 Max (0902)144 / 144018.14 s14.76 s$0.6158$4.2762
Muse Spark 1.3144 / 144004.22 s9.18 s$0.4342$3.0151
GPT-5.6 Luna144 / 144002.34 s4.80 s$0.0440$0.3057
Jev 1.14144 / 144000.37 s0.48 s$0.0159$0.1104
GPT-6 Luna Decisions72 / 72000.34 s0.90 s$0.0085$0.1177

Comprobaciones de presencia: la tarea común de sí/no

Compara el tiempo de respuesta y el coste en los mismos casos de presencia. Las solicitudes de diseño de los modelos generales también exigen una región, por lo que su carga de trabajo difiere de Decisions limitado a clasificación.

ModeloCasos de presenciaMedianaP95Coste de presencia por 1.000 solicitudes
GPT-6 Luna361.45 s1.88 s$0.0752
GPT-6.1 Sol362.18 s2.68 s$1.4487
Claude Sonnet 5.5361.85 s2.78 s$2.9955
Gemini 3.8 Flash362.45 s2.98 s$1.2008
Grok 4.7361.97 s5.95 s$2.8797
Qwen3.8 Max (0902)364.51 s6.79 s$2.6708
Muse Spark 1.3362.06 s4.23 s$2.0156
GPT-5.6 Luna361.58 s2.11 s$0.1533
Jev 1.14360.37 s0.48 s$0.1069
GPT-6 Luna Decisions360.31 s0.89 s$0.1062

Metodología

Qué ha cambiado y qué se ha mantenido fijo

  • Las seis capturas, anotaciones, instrucciones y esquemas de respuesta originales proceden del corpus del 13/08/2026. La nueva edición conserva sus propias copias.
  • Cada caso se ejecuta tres veces. Los modelos generativos solicitan un esfuerzo de razonamiento bajo y una salida de hasta 2.048 tokens, frente a los 400 anteriores. Los proveedores pueden declarar tokens de razonamiento facturables adicionales por encima de ese límite. Este cambio en las condiciones de ejecución se declara expresamente.
  • Las solicitudes se fijan a endpoints estándar operativos de los proveedores originales, sin rutas alternativas. Se conservan los endpoints, precios, ajustes y hashes exactos.
  • Los modelos de decisión reciben preguntas nativas junto con la tarea. Las probabilidades Noul usan un umbral fijo de 0,5. El punto de clic de Jev se calcula desde el centro del recuadro; los demás modelos con capacidad geométrica predicen un punto de clic separado.
  • Luna Decisions admite clasificación, pero no salida Box. Su evaluación de diseño mide únicamente la presencia y categoría de defectos. Las salidas tipadas nativas y la validez del JSON generado por un adaptador no equivalen a la generación libre de JSON.
  • Las respuestas inválidas no se corrigen ni se repiten. Solo los fallos transitorios de transporte permiten un reintento; se conservan todos los intentos. Las variaciones del proveedor y de la red pueden afectar los tiempos de respuesta. Los costes declarados incluyen descuentos automáticos de caché; las repeticiones posteriores pueden reutilizar cachés calientes.

Tres repeticiones miden la consistencia en estos casos. Seis pantallas sintéticas no demuestran superioridad general, fiabilidad en producción ni capacidad de navegación de varios pasos. No se evalúan acciones en dispositivos, recuperación ni consecución de objetivos de navegación.

Contrato de entrada

Solo capturas de pantalla

Texto del prompt más un PNG en base64: sin DOM, árbol de accesibilidad, código fuente, OCR, herramientas, plugins ni búsqueda web.

01

Renderizado fijo

Chromium a 1440 × 900 o 390 × 844; factor de escala del dispositivo: 1.

02

Geometría normalizada

Verdad de referencia derivada del navegador en un espacio de coordenadas de 0–1000.

03

Pares estables

Las variantes limpias y defectuosas conservan la geometría objetivo y los hashes de captura.

Operaciones

Mantén visible el comportamiento del proveedor

Identificadores de modelo fijados, rutas de proveedores originales, alternativas desactivadas y contratos de respuesta específicos de cada tarea documentados.

Política de reintentos

Un reintento para errores 429, 5xx o de red; no para respuestas incorrectas.

Métricas operativas

La latencia, los fallos, los reintentos y el coste se mantienen separados de la exactitud.

Privacidad

Los registros publicados omiten las claves y los identificadores de solicitud del proveedor.

Datos y licencia

Descargar esta edición

Comprobación previa del modelo ↓Plan de ejecución ↓Manifiesto de capturas ↓Resultados y puntuaciones individuales ↓Intentos originales conservados ↓Hallazgos e inventario de fallos ↓

Las capturas y resultados agregados creados por Repeato están disponibles bajo CC BY 4.0. Las salidas originales de los modelos siguen sujetas a las condiciones de los proveedores. Consulta el aviso de licencia.

ES