A chatbot answers. An AI agent acts: it calls tools, searches systems, and independently takes multiple steps to complete a task. That difference in autonomy calls for a different testing approach.
Where a language model previously only generated text, an agent today can create a ticket, initiate a payment, or send an email without a user intervening. That step from answering to acting is exactly where most existing testing approaches fall short.
A chatbot that gives a wrong answer is annoying. An agent that carries out a wrong action has already changed something before anyone could intervene: a wrong API call, a duplicate order, an email to the wrong recipient. That difference in consequences is the core reason why agents need their own testing approach.
An agent can call the right tool with the wrong parameters, or use a tool that wasn't needed for the task. Every call is a chance for an error that only becomes visible further down the chain.
An agent that hits an unexpected result can keep trying: calling the same tool repeatedly in hope of a different outcome. Without a hard limit on steps or costs, that builds up unnoticed.
An agent with access to a system sometimes uses that access more broadly than intended: a read permission that also allows write actions, or a task that goes beyond the original instruction.
For us, validating agentic AI systems is an extension of the same discipline we already apply to classic AI systems: not just checking whether the output is correct, but whether the system keeps acting within its limits, step by step.
At a client with an agent that updates customer records and sends confirmation emails, a test with a deliberately slow database got stuck: the agent received no confirmation of the write action, interpreted that as a failure, and retried the update three times. Result: three identical confirmation emails to the same customer.
No error message appeared, and no test failed in the classic sense. The agent did exactly what it was told: retry on failure. The problem was the absence of a retry limit and a check for whether the previous attempt may have actually succeeded.
The fix had two parts: a hard limit of one retry, and an idempotency check that first verifies whether the action was already carried out before attempting it again.
We test the entire chain: from individual tool calls to the boundaries an agent should never cross.
Book a call