AI is currently having a very good marketing year in test automation.
Almost every tool has become AI-powered. Sometimes this means genuinely useful new capabilities. Sometimes it means the old feature received a sparkle icon and was sent back onto the website wearing a futuristic jacket.
At the same time, general-purpose agents such as Claude or Codex are becoming surprisingly capable. They can write code, operate browsers, inspect screenshots, and keep working through a task for a long time without asking for a snack.
So there is an obvious and slightly uncomfortable question for companies building test automation tools:
Why not give the entire job to an AI agent?
I use coding agents every day. We are also adding AI capabilities to Repeato. I am very much not arguing that AI is useless.
But a successful agent demo and a production regression suite are two very different animals.
One is impressive for five minutes.
The other has to execute thousands of steps, finish before the next release, follow a reproducible path, and tell you exactly what went wrong when the app decides that Tuesday is an excellent day to break the checkout flow.
That difference is why we built the Repeato Vision Bench.
We wanted to measure where current vision-language models are already useful for UI testing—and where giving every click to an LLM would turn a fast test suite into a very intelligent waiting room.
A Demo Is Not a Regression Suite
An AI agent can already navigate many interfaces from a high-level instruction.
You can tell it:
Book a flight from Vienna to London and verify that the confirmation screen appears.
The agent may succeed.
That is impressive.
But which route did it take?
Perhaps it selected the dates first. Perhaps it chose the destination first. Perhaps it accepted a suggested airport, opened a fare comparison, dismissed an offer for extra luggage, or wandered into business class for a brief look at how the other half lives.
There can be several valid paths through the same interface.
For exploratory testing, this is useful. An agent that takes unexpected routes may find unexpected problems.
For regression testing, it creates a problem.
If the agent chooses a different route on every run, the runs are no longer directly comparable. A failure may mean:
- the application is broken,
- the agent selected a different path,
- the model interpreted the instruction differently,
- or the airline website has once again invented a new way to ask whether you want travel insurance.
A repeatable automated test should follow a known sequence, make assertions at meaningful checkpoints, and produce comparable evidence.
We can constrain the agent with more detailed instructions. But as those instructions become a precise list of actions and assertions, something interesting happens:
We have defined a test again.
The unsolved question is not whether an AI agent can click through a UI.
It is whether it can do so with the speed, control, and reproducibility expected from a serious regression suite.
Ten Seconds Is Fast Until You Multiply It by 1,000
Model response time is easy to underestimate.
If one AI-driven step takes 10 seconds, nobody panics. You look away, take a sip of coffee, and the answer is there.
If 1,000 dependent steps each take 10 seconds, the coffee has become a lifestyle.
In the published grounding run of our benchmark, average wall-clock time ranged from approximately 1.7 to 11.9 seconds per retained case, depending on the model configuration.
Now imagine that 1,000 serial UI interactions each require a comparable model call.
The model waiting time alone would add up to roughly:
- 28 minutes at 1.7 seconds per step;
- 3 hours and 18 minutes at 11.9 seconds per step.
This is an illustration, not an end-to-end suite benchmark. A real test run also includes application startup, animations, device communication, backend requests, and the occasional loading spinner that appears to be reconsidering its career.
Independent tests can also run in parallel.
But inside one workflow, step 43 often depends on step 42. The agent cannot confirm the booking before it has selected the flight, no matter how motivational the prompt is.
Agent workflows may add still more time for reasoning and tool calls.
This makes latency more than a usability detail. It becomes an architectural constraint.
The goal cannot be to attach an AI request to every tap because AI is fashionable.
The goal is to spend AI latency where the additional understanding is actually worth it.
So We Built a Benchmark
We had a working assumption inside Repeato:
Large vision-language models are becoming good at understanding screens, but they are still too slow and too variable for the high-frequency interaction loop of test automation.
Assumptions are useful. Numbers are more useful. Numbers are also less likely to nod politely during a meeting and then betray you in production.
The current Vision Bench pilot evaluates nine vision-capable model configurations across three task families:
- Grounding: Given a screenshot and a natural-language target, return the element’s bounding box.
- Presence: Decide whether a described UI element is present.
- Defect classification: Decide whether a screenshot contains a visible layout defect and identify its category.
For every retained request, we store the raw response and normalized prediction. We then measure task quality, schema validity, wall-clock latency, and estimated API cost.
This gives us a practical basis for deciding which jobs an LLM should perform inside Repeato.
It does not measure a complete autonomous agent. It also does not compare a complete Repeato test run with a complete agent test run.
This first release is a pilot: 24 grounding cases, 12 presence cases, and 12 defect-classification cases.
That is enough to expose useful engineering trade-offs.
It is not enough to declare one model the emperor of all pixels.
Grounding Works—But “Returned a Box” Is Not the Same as “Clicked Correctly”
The best grounding result in this run came from GPT-5.6 Luna.
It returned a valid box for all 24 cases and achieved a mean Intersection over Union (IoU) of 0.771.
IoU measures how closely the predicted box overlaps the annotated target. A perfect result is 1.0.
This is a strong result for a general-purpose model.
It also shows why the details matter.
A model can return a perfectly valid JSON object containing a perfectly valid rectangle that is perfectly positioned over the wrong button.
Schema compliance is not interaction accuracy.
Small controls, visually dense screens, and ambiguous descriptions remain harder than large obvious targets. If the returned point misses the button by a few pixels, the model has not almost passed the test. It has clicked something else.
User interfaces can be rather unforgiving about philosophical victories.
Presence Detection Is Already Very Useful
Eight of the nine tested configurations classified all 12 presence cases correctly.
The sample is small, and the current task is close to saturation. Still, this is a promising result.
Presence checks are a natural job for AI because they ask a semantic question:
Is the thing I mean visible on this screen?
That is different from asking a model to control every movement of the test.
The model gets to use its broader visual understanding for one valuable decision. The fast execution engine can handle the routine work before and after it.
This is a much healthier division of labor than sending every swipe to a remote model and hoping the network is feeling athletic.
Finding a Defect and Naming It Are Different Skills
On the defect task, Grok 4.5 achieved 100% balanced accuracy when deciding whether a defect was present.
Its defect-category accuracy was 66.7%.
In other words, the model was excellent at saying, “Something is wrong here.”
It was less consistent at explaining exactly what kind of wrongness had occurred.
This distinction matters in production.
A broad screen-integrity check can still be extremely useful. It can flag a suspicious result and preserve the screenshot for review.
But if a workflow depends on the precise diagnosis—overlap, clipping, displacement, sizing, or another category—the label may need additional evidence or human confirmation.
This is a recurring theme in applied AI: detection often arrives before dependable explanation. The model sees the smoke. It is still filling out the fire report.
Accuracy Alone Does Not Choose the Production Model
Benchmark leaderboards invite us to look for one winner.
Production systems are less cooperative.
The best model for a particular test step depends on:
- task accuracy;
- response latency;
- cost per run and release cycle;
- invalid or unusable response rates;
- reproducibility across repeated executions;
- and the damage caused by a false positive, false negative, or inaccurate click.
A model can be accurate but too slow.
It can be fast but expensive.
It can be cheap but occasionally return an answer shaped like JSON’s distant cousin.
The grounding trade-off chart uses Box IoU for accuracy, total cost for 24 requests, and color for mean response latency. Higher and further left is better.
That is why our benchmark records quality, latency, reliability, and cost separately.
We are not searching for the smartest model in the abstract.
We are trying to select the right mechanism for a specific job.
Where Fast Computer Vision Still Wins
Repeato creates visual fingerprints for recorded targets and can use view IDs or the application’s layout hierarchy as fallback strategies.
The local computer-vision loop can inspect the screen up to 20 times per second. It does not need to upload a screenshot, wait for remote inference, and download a small rectangle describing where a button used to be.
That speed matters for ordinary test execution.
It matters even more when an element is moving.
A game control, animation, map marker, or transient interface state may change position while a model is still thinking. By the time the answer arrives, the button may have moved, completed its side quest, and sent a postcard.
Repeato can disable its normal animation wait and interact with moving targets using local visual matching.
Current large vision-language models are generally too slow to sit inside that control loop for every interaction.
The benchmark does not directly measure Repeato’s local matching speed against the models. It measures model requests on visual test primitives.
But the seconds-per-request results support the architectural concern: repeated, known interactions need a specialized and predictable execution path.
Where AI Already Earns Its Keep
There are also test questions that traditional matching cannot answer well.
For example:
- Is this graph trending upward?
- Are there three items in the basket?
- Does the displayed total make sense?
- Does this screen contain an obvious visual inconsistency?
These are semantic questions.
They are exactly where a vision-language model can add something genuinely new.
Repeato now includes a dedicated Check Screen Integrity step under Check content → Check for screen integrity. It complements Repeato’s other visual and AI-assisted testing features.
It allows a QA to add an automatic AI-based consistency review of the current interface to a test flow.
This is the kind of AI integration we find useful: one purposeful model call that interprets the whole screen, surrounded by fast and reproducible automation.
AI can also help outside the execution loop:
- exploring an unfamiliar flow;
- suggesting assertions;
- summarizing failures;
- comparing screenshots;
- and prioritizing evidence for review.
These tasks benefit from interpretation and tolerate a little more latency.
Nobody needs a philosophical essay before the test runner taps “Continue.”
But a thoughtful explanation after a failed batch can save a human a great deal of time.
Machine-Readable Reports Are Part of the Answer
If AI agents are going to help diagnose test failures, reports need to work for machines as well as humans.
A useful report should preserve:
- the route the test took;
- every step outcome;
- screenshots and timing;
- assertion evidence;
- and the exact point where actual behavior diverged from expectation.
Without that structure, asking an AI to diagnose a failed run is like inviting a detective to a crime scene after somebody has cleaned the room and left one blurry photo on the table.
Repeato keeps tests and results in open formats (.json and .jpeg) in a workspace directory on the user’s file system.
It can also export visual step-by-step PDF reports and searchable HTML batch reports. Repeato CLI writes execution artifacts and JUnit-compatible outputs.
Continuing to strengthen structured, machine-readable evidence is an important part of making AI-assisted diagnosis dependable.
It also does not require turning test execution into a hosted cloud service. Repeato remains local-first; cloud features are optional.
Better reports do not solve the reproducibility problem of an unconstrained agent.
But they do make a hybrid system observable. A later agent can inspect what actually happened instead of inventing a confident bedtime story about it.
The Practical Answer Is a Hybrid
The useful question is not:
Should test automation use AI?
Of course it should, where AI helps.
The useful question is:
Which mechanism should own each part of the job?
Our current answer is:
- use fast, repeatable computer vision and conventional locators for known interactions;
- use AI vision for semantic checks and ambiguous visual interpretation;
- use agents for exploration, assistance, and failure analysis where variability is acceptable;
- and keep the executed path and evidence structured enough to reproduce every important result.
The boundary will move.
Models will become faster, cheaper, and more consistent. Agent frameworks will improve. Some tasks that need specialized automation today may eventually be handled reliably by a general agent.
When that happens, excellent.
Our job is not to defend an old architecture from the future.
Our job is to build the fastest and most dependable test automation system available with the technology that actually exists today.
That requires both intelligence and restraint.
Or, put differently:
Use AI where it can understand something.
Do not make it supervise every pixel just because it owns a very impressive jacket.
Inspect the Evidence
The Vision Bench publication includes the methodology, frozen captures, annotations, manifests, sanitized model records, and aggregate results needed to inspect the published findings.
- Benchmark overview
- Methodology and limitations
- Machine-readable grounding results
- Machine-readable presence and defect results
- How Repeato combines local vision and AI
If you compare another model or prompting strategy, publish the exact configuration and raw outputs.
Reproducible evidence is more useful than a leaderboard screenshot.
It is also the standard we want AI-assisted test automation to meet.
If that hybrid approach fits the way your team tests, explore Repeato’s features or download the free version and try it on your own app.