# Vision pilot — 2026-10-17 Completed 1368/1368 case executions in 1368 HTTP attempts. Ten configurations; three repetitions; Claude Opus 5.5 excluded. Total reported cost: **$3.380247**. | Configuration | Target hits | Mean target-area overlap | Presence correct | Clean/broken correct | Exact category | Median / P95 seconds | Cost USD | |---|---:|---:|---:|---:|---:|---:|---:| | GPT-6 Luna | 71/72 | 79.6% | 36/36 | 33/36 | 10/18 | 2.177 / 3.825 | 0.018656 | | GPT-6.1 Sol | 72/72 | 81.2% | 36/36 | 33/36 | 6/18 | 2.675 / 4.881 | 0.280336 | | Claude Sonnet 5.5 | 70/72 | 74.8% | 36/36 | 30/36 | 9/18 | 2.204 / 3.989 | 0.580272 | | Gemini 3.8 Flash | 72/72 | 73.3% | 36/36 | 28/36 | 13/18 | 2.425 / 10.958 | 0.265489 | | Grok 4.7 | 52/72 | 47.7% | 36/36 | 34/36 | 13/18 | 12.026 / 28.382 | 1.117146 | | Qwen3.8 Max (0902) | 72/72 | 76.0% | 36/36 | 34/36 | 7/18 | 8.135 / 14.758 | 0.615780 | | Muse Spark 1.3 | 68/72 | 63.0% | 36/36 | 32/36 | 10/18 | 4.224 / 9.178 | 0.434180 | | GPT-5.6 Luna | 71/72 | 78.6% | 36/36 | 33/36 | 12/18 | 2.341 / 4.799 | 0.044016 | | Jev 1.14 | 72/72 | 58.1% | 36/36 | 36/36 | 12/18 | 0.365 / 0.483 | 0.015900 | | GPT-6 Luna Decisions | Not supported | — | 36/36 | 30/36 | 9/18 | 0.338 / 0.898 | 0.008471 | ## Shared presence-task operations | Configuration | Presence cases | Median / P95 seconds | Cost per 1,000 presence requests, USD | |---|---:|---:|---:| | GPT-6 Luna | 36 | 1.450 / 1.882 | 0.075247 | | GPT-6.1 Sol | 36 | 2.181 / 2.677 | 1.448675 | | Claude Sonnet 5.5 | 36 | 1.849 / 2.776 | 2.995500 | | Gemini 3.8 Flash | 36 | 2.447 / 2.978 | 1.200792 | | Grok 4.7 | 36 | 1.967 / 5.955 | 2.879667 | | Qwen3.8 Max (0902) | 36 | 4.514 / 6.793 | 2.670833 | | Muse Spark 1.3 | 36 | 2.060 / 4.227 | 2.015586 | | GPT-5.6 Luna | 36 | 1.582 / 2.107 | 0.153295 | | Jev 1.14 | 36 | 0.368 / 0.479 | 0.106892 | | GPT-6 Luna Decisions | 36 | 0.306 / 0.893 | 0.106208 | Overlap means intersection over union (IoU). Invalid and failed attempts remain in task denominators. Each defect-category denominator contains six distinct defects repeated three times, not 18 independent defects. Luna Decisions has half as many cases; total cost is not an equal-workload cost comparison. The page also reports cost per 1,000 HTTP requests. ## Failures and consistency | Configuration | Invalid final responses | Final transport failures | HTTP attempts | Targets with consistent hit/miss across three repeats | |---|---:|---:|---:|---:| | GPT-6 Luna | 0 | 0 | 144 | 23/24 | | GPT-6.1 Sol | 0 | 0 | 144 | 24/24 | | Claude Sonnet 5.5 | 8 | 0 | 144 | 22/24 | | Gemini 3.8 Flash | 1 | 0 | 144 | 24/24 | | Grok 4.7 | 2 | 0 | 144 | 16/24 | | Qwen3.8 Max (0902) | 1 | 0 | 144 | 24/24 | | Muse Spark 1.3 | 0 | 0 | 144 | 23/24 | | GPT-5.6 Luna | 0 | 0 | 144 | 23/24 | | Jev 1.14 | 0 | 0 | 144 | 24/24 | | GPT-6 Luna Decisions | 0 | 0 | 72 | Not supported | Consistency can mean consistently wrong. Invalid predictions are never repaired or rerun; only transient transport failures receive one retry. Every HTTP attempt is retained and contributes to reported cost. ## Case review inventory ### GPT-6 Luna Returned versions: `openai/gpt-6-luna`. - [banking-ios/defect/layout-defect](pilot-captures/banking-ios/defect.png): affected repeats 1, 2, 3; success. 1 incorrect classifications; 3 incorrect categories. Example finish reason: stop. - [smart-home-android/defect/layout-defect](pilot-captures/smart-home-android/defect.png): affected repeats 1, 2, 3; success. 1 incorrect classifications; 3 incorrect categories. Example finish reason: stop. - [flight-pass-ios/clean/grounding/profile-tab](pilot-captures/flight-pass-ios/clean.png): affected repeats 2; success. 1 listed target misses. Example finish reason: stop. - [commerce-catalog-desktop/defect/layout-defect](pilot-captures/commerce-catalog-desktop/defect.png): affected repeats 3; success. 0 incorrect classifications; 1 incorrect categories. Example finish reason: stop. - [iot-control-desktop/defect/layout-defect](pilot-captures/iot-control-desktop/defect.png): affected repeats 3; success. 1 incorrect classifications; 1 incorrect categories. Example finish reason: stop. ### GPT-6.1 Sol Returned versions: `openai/gpt-6.1-sol`. - [analytics-dashboard-desktop/defect/layout-defect](pilot-captures/analytics-dashboard-desktop/defect.png): affected repeats 1, 2, 3; success. 0 incorrect classifications; 3 incorrect categories. Example finish reason: stop. - [iot-control-desktop/defect/layout-defect](pilot-captures/iot-control-desktop/defect.png): affected repeats 1, 2, 3; success. 3 incorrect classifications; 3 incorrect categories. Example finish reason: stop. - [banking-ios/defect/layout-defect](pilot-captures/banking-ios/defect.png): affected repeats 1, 2, 3; success. 0 incorrect classifications; 3 incorrect categories. Example finish reason: stop. - [smart-home-android/defect/layout-defect](pilot-captures/smart-home-android/defect.png): affected repeats 1, 2, 3; success. 0 incorrect classifications; 3 incorrect categories. Example finish reason: stop. ### Claude Sonnet 5.5 Returned versions: `anthropic/claude-sonnet-5.5`. - [commerce-catalog-desktop/clean/grounding/add-product](pilot-captures/commerce-catalog-desktop/clean.png): affected repeats 1; invalid_response. 1 listed target misses. Example finish reason: stop. - [commerce-catalog-desktop/defect/layout-defect](pilot-captures/commerce-catalog-desktop/defect.png): affected repeats 1, 3; invalid_response. 2 incorrect classifications; 2 incorrect categories. Example finish reason: stop. - [analytics-dashboard-desktop/defect/layout-defect](pilot-captures/analytics-dashboard-desktop/defect.png): affected repeats 1, 2, 3; success. 0 incorrect classifications; 3 incorrect categories. Example finish reason: stop. - [banking-ios/defect/layout-defect](pilot-captures/banking-ios/defect.png): affected repeats 1, 2; invalid_response. 2 incorrect classifications; 2 incorrect categories. Example finish reason: stop. - [smart-home-android/defect/layout-defect](pilot-captures/smart-home-android/defect.png): affected repeats 1, 3; invalid_response. 2 incorrect classifications; 2 incorrect categories. Example finish reason: stop. - [flight-pass-ios/clean/grounding/trips-tab](pilot-captures/flight-pass-ios/clean.png): affected repeats 3; invalid_response. 1 listed target misses. Example finish reason: stop. ### Gemini 3.8 Flash Returned versions: `google/gemini-3.8-flash`. - [analytics-dashboard-desktop/defect/layout-defect](pilot-captures/analytics-dashboard-desktop/defect.png): affected repeats 1, 2, 3; success. 0 incorrect classifications; 3 incorrect categories. Example finish reason: stop. - [banking-ios/clean/layout-defect](pilot-captures/banking-ios/clean.png): affected repeats 1, 2, 3; success. 3 incorrect classifications; 0 incorrect categories. Example finish reason: stop. - [flight-pass-ios/clean/layout-defect](pilot-captures/flight-pass-ios/clean.png): affected repeats 1, 2, 3; success, invalid_response. 3 incorrect classifications; 0 incorrect categories. Example finish reason: stop. - [flight-pass-ios/defect/layout-defect](pilot-captures/flight-pass-ios/defect.png): affected repeats 1, 2; success. 0 incorrect classifications; 2 incorrect categories. Example finish reason: stop. - [iot-control-desktop/clean/layout-defect](pilot-captures/iot-control-desktop/clean.png): affected repeats 2; success. 1 incorrect classifications; 0 incorrect categories. Example finish reason: stop. - [smart-home-android/clean/layout-defect](pilot-captures/smart-home-android/clean.png): affected repeats 3; success. 1 incorrect classifications; 0 incorrect categories. Example finish reason: stop. ### Grok 4.7 Returned versions: `x-ai/grok-4.7`. - [commerce-catalog-desktop/defect/layout-defect](pilot-captures/commerce-catalog-desktop/defect.png): affected repeats 1; invalid_response. 1 incorrect classifications; 1 incorrect categories. Example finish reason: stop. - [analytics-dashboard-desktop/clean/grounding/compare-toggle](pilot-captures/analytics-dashboard-desktop/clean.png): affected repeats 1, 2, 3; success. 3 listed target misses. Example finish reason: stop. - [banking-ios/clean/grounding/profile](pilot-captures/banking-ios/clean.png): affected repeats 1, 2, 3; success. 3 listed target misses. Example finish reason: stop. - [flight-pass-ios/defect/layout-defect](pilot-captures/flight-pass-ios/defect.png): affected repeats 1, 2, 3; success. 0 incorrect classifications; 3 incorrect categories. Example finish reason: stop. - [smart-home-android/clean/grounding/more](pilot-captures/smart-home-android/clean.png): affected repeats 1, 2, 3; success. 3 listed target misses. Example finish reason: stop. - [smart-home-android/clean/grounding/add-device](pilot-captures/smart-home-android/clean.png): affected repeats 1; success. 1 listed target misses. Example finish reason: stop. - [commerce-catalog-desktop/clean/grounding/stock-filter](pilot-captures/commerce-catalog-desktop/clean.png): affected repeats 2, 3; success. 2 listed target misses. Example finish reason: stop. - [analytics-dashboard-desktop/clean/grounding/search-field](pilot-captures/analytics-dashboard-desktop/clean.png): affected repeats 2; success. 1 listed target misses. Example finish reason: stop. - [iot-control-desktop/clean/grounding/auto-mode](pilot-captures/iot-control-desktop/clean.png): affected repeats 2; success. 1 listed target misses. Example finish reason: stop. - [banking-ios/clean/grounding/hide-balance](pilot-captures/banking-ios/clean.png): affected repeats 2, 3; success. 2 listed target misses. Example finish reason: stop. - [banking-ios/clean/layout-defect](pilot-captures/banking-ios/clean.png): affected repeats 2; success. 1 incorrect classifications; 0 incorrect categories. Example finish reason: stop. - [flight-pass-ios/clean/grounding/check-in](pilot-captures/flight-pass-ios/clean.png): affected repeats 2, 3; success. 2 listed target misses. Example finish reason: stop. - [smart-home-android/defect/layout-defect](pilot-captures/smart-home-android/defect.png): affected repeats 2; success. 0 incorrect classifications; 1 incorrect categories. Example finish reason: stop. - [commerce-catalog-desktop/clean/grounding/add-product](pilot-captures/commerce-catalog-desktop/clean.png): affected repeats 3; invalid_response. 1 listed target misses. Example finish reason: stop. - [smart-home-android/clean/grounding/thermostat-card](pilot-captures/smart-home-android/clean.png): affected repeats 3; success. 1 listed target misses. Example finish reason: stop. ### Qwen3.8 Max (0902) Returned versions: `qwen/qwen3.8-max-0902`. - [commerce-catalog-desktop/defect/layout-defect](pilot-captures/commerce-catalog-desktop/defect.png): affected repeats 1, 3; success. 0 incorrect classifications; 2 incorrect categories. Example finish reason: stop. - [analytics-dashboard-desktop/defect/layout-defect](pilot-captures/analytics-dashboard-desktop/defect.png): affected repeats 1, 2; success. 0 incorrect classifications; 2 incorrect categories. Example finish reason: stop. - [iot-control-desktop/clean/layout-defect](pilot-captures/iot-control-desktop/clean.png): affected repeats 1; invalid_response. 1 incorrect classifications; 0 incorrect categories. Example finish reason: stop. - [iot-control-desktop/defect/layout-defect](pilot-captures/iot-control-desktop/defect.png): affected repeats 1; success. 1 incorrect classifications; 1 incorrect categories. Example finish reason: stop. - [flight-pass-ios/defect/layout-defect](pilot-captures/flight-pass-ios/defect.png): affected repeats 1, 2, 3; success. 0 incorrect classifications; 3 incorrect categories. Example finish reason: stop. - [smart-home-android/defect/layout-defect](pilot-captures/smart-home-android/defect.png): affected repeats 1, 2, 3; success. 0 incorrect classifications; 3 incorrect categories. Example finish reason: stop. ### Muse Spark 1.3 Returned versions: `meta/muse-spark-1.3`. - [iot-control-desktop/defect/layout-defect](pilot-captures/iot-control-desktop/defect.png): affected repeats 1, 2, 3; success. 3 incorrect classifications; 3 incorrect categories. Example finish reason: stop. - [banking-ios/clean/grounding/hide-balance](pilot-captures/banking-ios/clean.png): affected repeats 1; success. 1 listed target misses. Example finish reason: stop. - [flight-pass-ios/clean/grounding/boarding-pass](pilot-captures/flight-pass-ios/clean.png): affected repeats 1, 2, 3; success. 3 listed target misses. Example finish reason: stop. - [flight-pass-ios/defect/layout-defect](pilot-captures/flight-pass-ios/defect.png): affected repeats 1, 2, 3; success. 0 incorrect classifications; 3 incorrect categories. Example finish reason: stop. - [smart-home-android/defect/layout-defect](pilot-captures/smart-home-android/defect.png): affected repeats 1, 2; success. 1 incorrect classifications; 2 incorrect categories. Example finish reason: stop. ### GPT-5.6 Luna Returned versions: `openai/gpt-5.6-luna`. - [iot-control-desktop/defect/layout-defect](pilot-captures/iot-control-desktop/defect.png): affected repeats 1, 2; success. 2 incorrect classifications; 2 incorrect categories. Example finish reason: stop. - [banking-ios/clean/grounding/transaction-search](pilot-captures/banking-ios/clean.png): affected repeats 1; success. 1 listed target misses. Example finish reason: stop. - [commerce-catalog-desktop/defect/layout-defect](pilot-captures/commerce-catalog-desktop/defect.png): affected repeats 2; success. 0 incorrect classifications; 1 incorrect categories. Example finish reason: stop. - [banking-ios/defect/layout-defect](pilot-captures/banking-ios/defect.png): affected repeats 2, 3; success. 0 incorrect classifications; 2 incorrect categories. Example finish reason: stop. - [smart-home-android/defect/layout-defect](pilot-captures/smart-home-android/defect.png): affected repeats 2; success. 1 incorrect classifications; 1 incorrect categories. Example finish reason: stop. ### Jev 1.14 Returned versions: `typesafe/jev-1.14-20261014`. - [commerce-catalog-desktop/defect/layout-defect](pilot-captures/commerce-catalog-desktop/defect.png): affected repeats 1, 2, 3; success. 0 incorrect classifications; 3 incorrect categories. - [flight-pass-ios/defect/layout-defect](pilot-captures/flight-pass-ios/defect.png): affected repeats 1, 2, 3; success. 0 incorrect classifications; 3 incorrect categories. ### GPT-6 Luna Decisions Returned versions: `openai/gpt-6-luna-decisions-20261006`. - [commerce-catalog-desktop/defect/layout-defect](pilot-captures/commerce-catalog-desktop/defect.png): affected repeats 1, 2, 3; success. 3 incorrect classifications; 3 incorrect categories. - [banking-ios/defect/layout-defect](pilot-captures/banking-ios/defect.png): affected repeats 1, 2, 3; success. 3 incorrect classifications; 3 incorrect categories. - [flight-pass-ios/defect/layout-defect](pilot-captures/flight-pass-ios/defect.png): affected repeats 1, 2, 3; success. 0 incorrect classifications; 3 incorrect categories. ## Methodology and interpretation - Frozen images and annotations come from 2026-08-13. Every task retains image, prompt, schema, and execution-contract hashes. Original editions remain unchanged. - Generative requests request low reasoning effort and a 2,048-token output allowance, versus 400 in the original pilot. Providers can report additional billed reasoning tokens beyond that allowance (observed in this run). This is a disclosed configuration change, so differences do not isolate model upgrades alone. - Jev predicts a Box, and the adapter derives its centre as the click point. Other geometry-capable models independently predict the click point. These are different point semantics. - Luna Decisions supports classification without Box output. Its layout contract evaluates issue presence and category, without requiring a region. Unsupported geometry is not counted as a failed coordinate prediction. - Native question adapters differ from free JSON generation. Adapter-created schema validity is not a measure of the model's free-text JSON reliability. Noul uses a fixed, untuned 0.5 threshold. - The existing defect scorer assigns zero region overlap when the category is wrong. Schema-invalid general-model responses count as failures even if some returned fields could have been useful. - Timing includes HTTP round trip and local parsing for final retained attempts, with no metadata polling. Cost includes every attempt and any automatic caching discounts; later repetitions can have warmer caches. - Three repetitions on six synthetic screens measure repeatability, not additional diversity. No device action, multi-step navigation, recovery, or navigation-goal completion is tested. ⚠️ Bias Alert: Newer model names and more repetitions do not establish broader coverage. Near-perfect pilot scores can coexist with poor real-app reliability. 👿 Skeptical assessment: The weakest assumption is that six controlled scenes represent customers' real interfaces and failure modes. The edition guides further validation; it does not establish an overall model ranking or a production-ready navigation agent. ## Evidence - [All aggregate and per-case scores](results.json) - [Sanitized retained attempts](raw/attempts.jsonl) - [Pinned endpoints, prices, settings](preflight.json) - [Frozen run plan](plan.json) - [Capture manifest](pilot-captures/manifest.json) Edition page: /vision-bench/editions/2026-10-17/. The top selector switches back to the original pilot.