What AI testing still cannot do today

Most “AI testing tools” are a GPT wrapper with a nice interface. A useful filter to separate what genuinely works from conference-demo magic.

After 25 years of QA podcasts and hundreds of interviews with test leaders, despite a flood of "agentic QA" marketing, one observation still holds up. That's not cynicism: it's a useful starting point for determining what a tool actually solves.

What is production-ready today

// In production at serious teams

Self-healing locatorsyes, with human oversight

Test generation from specificationsyes, as a starting point

Fully autonomous testing without oversightno, not yet

Teams at large technology companies that use AI in production for testing are notably selective: they deploy AI on specific, well-defined problems (flaky-test detection, selector repair, test generation for Playwright), not as a replacement for their entire testing approach.

AI takes over the repetitive, exhausting work that drives good engineers away. What it doesn't do: understand which business risk matters, or make a judgment call when a test result is ambiguous.

Three signals you're looking at hype, not a product

"Our AI tests everything autonomously, with no humans"

This kind of claim rarely survives its own proof of concept. Always ask for a demo on your own application, not the vendor's polished showcase.

Benchmarks without methodology

"90% less maintenance" says nothing without knowing on which application, under what conditions, and compared to what. Self-healing benchmarks are rarely comparable between vendors.

No visibility into what the system actually did

A tool that makes changes without a log or review option builds up technical debt you only discover later.

What does work, in the right order

Start from the pain, not the hype. Are you losing time to flaky tests in CI/CD? Does test maintenance cost a day per sprint? Is coverage missing to release with confidence? Then look for the tool that solves that specific problem, not the platform that promises to do everything at once. A focused solution that genuinely solves one problem delivers more than a complete platform that contributes a little bit everywhere.

We evaluate AI testing tools exactly that way: not based on the marketing claim, but based on what they actually solve on your own suite.

An example from practice

A vendor demonstrated a "fully autonomous AI tester" that would build and maintain an entire regression suite without human intervention. On the vendor's own demo application, it worked flawlessly. On a client's suite, with a slightly more complex multi-step checkout, the tool failed at step three of five: it couldn't reliably distinguish between an intermediate loading screen and an actual error.

// What the proof-of-concept exposed

On the vendor's demo applicationworked as promised

On the client's own checkout flowfailed at step 3 of 5

Fix requiredmanual configuration per step

That's not an exception, it's the rule: a tool that works perfectly on a controlled demo says little about how it performs on an application with its own quirks. The only way to see that difference is to test on your own suite, not the vendor's.

A practical question list for a vendor conversation

"Can I test this on my own application, not your demo?" A vendor who hesitates here has already answered the question.
"What happens when the system gets it wrong?" A good answer describes a concrete review process; a vague one ("that rarely happens") is a warning sign.
"Can I see and roll back every automatic change?" Without a log and rollback, you're building trust on a black box.

// No hype, just results

Not sure which AI testing tools would actually deliver value for you?

We evaluate tools on your own suite, not the vendor's showcase.

Book a call