Grounding scenario · 2026-08-13
Every model received the same screenshot, prompt, and structured response contract.
390 × 844 · normalized 0–1000 coordinates
Exact input
You are evaluating a user interface screenshot for test automation.
Locate the single UI element described below. Return the best click point and the tight visible bounding box of the complete interactive target. Coordinates use a normalized 0 to 1000 grid, where (0, 0) is the top-left image corner and (1000, 1000) is the bottom-right image corner.
Target: the Trips bottom tab
Return only data matching the supplied JSON schema. Do not describe your reasoning.
ExpectedModel evidence
Blue marks the browser-derived target. Orange marks the model’s returned box and point. Format validity checks only the response contract; it does not mean the requested element was found.
{"point": {"x": 174, "y": 961}, "box": {"x": 139, "y": 940, "width": 100, "height": 60}}{"point": {"x": 187, "y": 953}, "box": {"x": 120, "y": 920, "width": 140, "height": 70}}{"point":{"x":66,"y":812},"box":{"x":0,"y":780,"width":131,"height":64}}{
"point": {
"x": 185,
"y": 962
},
"box": {
"x": 150,
"y": 935,
"width": 70,
SyntaxError
{"point":{"x":167,"y":963},"box":{"x":0,"y":925,"width":333,"height":75}}{"point":{"x":185,"y":975},"box":{"x":154,"y":944,"width":62,"height":39}}{"point":{"x":72,"y":817},"box":{"x":47,"y":793,"width":50,"height":38}}{"point":{"x":181,"y":2107},"box":{"x":26,"y":2035,"width":310,"height":144}}point must use normalized 0..1000 coordinates. box must be positive and contained in normalized 0..1000 coordinates.
{"point": {"x": 182, "y": 962}, "box": {"x": 155, "y": 943, "width": 57, "height": 41}}