You change one line in a prompt. The demo still looks great. Two days later, a customer support agent starts issuing refunds when users only asked to pause their subscription. Nothing crashed, no error was logged, and every unit test passed.
This is the new reality of building with large language models. AI agents now read documents, call APIs, update databases and talk to real customers. But the old testing playbook assumes code is deterministic: same input, same output. LLMs break that assumption. The same question can produce different answers, and a model upgrade from your provider can change behaviour overnight without a single line of your code changing.
In 2026, almost every product roadmap has an AI feature on it, and most teams still test those features by “trying a few prompts and seeing if it looks right.” That works for a prototype. It does not work when the agent can touch money, data or customer trust.
This blog explains a practical, layered framework for testing AI agents and LLM apps, with code examples and lessons from real production work. It is written for developers, tech leads and product owners who are shipping (or about to ship) AI features and want them to be reliable, safe and affordable.
The Problem: Why Traditional Testing Fails for AI
A real-world scenario
Imagine an e-commerce brand that runs on subscriptions. They add an AI support agent that can look up orders, answer product questions from a knowledge base, and perform actions like pausing, skipping or cancelling a subscription. In the first week, it handles hundreds of conversations and the team is delighted.
Then the problems start to appear quietly. The agent invents a return policy that doesn’t exist. It calls the cancel tool when the user said “skip next month.” It gets stuck in a loop calling the order lookup tool five times. None of this shows up in logs as an error, because technically everything “worked.”
Business impact
- Lost revenue: wrong actions like accidental cancellations or refunds cost real money.
- Trust damage: one confidently wrong answer screenshotted on social media can undo months of brand work.
- Hidden cost: agents that loop or use oversized prompts burn tokens, and the API bill grows faster than usage.
- Slow releases: without tests, every prompt change feels risky, so teams stop improving the product.
Technical challenges
- Non-determinism: the same input can produce different outputs, so exact-match assertions fail.
- No single correct answer: “Is this a good answer?” is a judgment, not a string comparison.
- Multi-step behaviour: an agent can reach the right final answer through a wrong or dangerous path.
- Moving foundations: model versions, retrieval data and prompts all change independently.
The Solution: A Layered Testing Framework
The approach that works is the same idea as the classic testing pyramid, adapted for AI. Run many cheap, fast checks at the bottom, and fewer expensive, realistic checks at the top. Each layer catches a different type of failure.
Layer 1: Deterministic checks (every commit)
A surprising amount of an AI app is not AI at all. Tool schemas, argument validation, output formats and guardrails can be tested with normal unit tests. We mock the LLM response and test what our code does with it. These tests run in seconds and should never be skipped.
// tools.test.ts (Vitest)
it('rejects cancel when the user only asked to skip', () => {
const call = { name: 'cancel_subscription', args: { id: 'sub_123' } };
const result = guardrail.validate(call, { intent: 'skip' });
expect(result.allowed).toBe(false);
expect(result.requiresConfirmation).toBe(true);
});Layer 2: Evaluation datasets with LLM-as-judge (every PR)
This is the heart of the framework. We build a golden dataset: 50 to 300 real or realistic user inputs, each with the expected behaviour (which tool should be called, which facts must appear, what must never be said). On every pull request, the agent runs against this dataset and each output is scored.
Some scores are simple rules (did it call the right tool?). Others need judgment (is the answer grounded in the retrieved documents?). For those, a second model acts as a judge, using a strict rubric and returning a structured score.
// judge.ts - grounding scorer
const verdict = await judge.messages.create({
model: JUDGE_MODEL,
temperature: 0,
system: 'You grade answers. Reply only with JSON: {"grounded": bool, "reason": string}',
messages: [{ role: 'user', content:
`CONTEXT:\n${retrievedDocs}\n\nANSWER:\n${agentAnswer}\n\n` +
'Is every factual claim in ANSWER supported by CONTEXT?' }],
});Key decision: we compare scores against the last accepted baseline, not against 100%. A PR fails if grounding drops by more than 3 points or if any “critical” case (like refunds or cancellations) fails at all.
Layer 3: Agent trajectory tests (nightly)
For agents, the final answer is not enough. We record the full trajectory: every tool call, its arguments and the order. Then we assert things like “never call cancel_subscription without a confirmation step” or “resolve this in five steps or fewer.” Because these runs are slower and cost more tokens, they run nightly on the full dataset, and each case runs three times to measure consistency.
Layer 4: Production monitoring (always on)
No dataset covers everything real users do. We log traces of live conversations, sample a percentage for automatic judging, and track thumbs-down feedback. Every real failure becomes a new test case. This feedback loop is what makes the system stronger every week.
Tools we use
- Test runners: Vitest / Jest for TypeScript, PHPUnit / Pest for Laravel backends.
- Models: Anthropic Claude API, OpenAI and AWS Bedrock, depending on the client’s stack.
- Eval and tracing: an open-source eval framework (such as Promptfoo or DeepEval) or a small custom harness, plus a tracing tool (such as Langfuse) for production logs.
- CI: GitHub Actions, with eval results posted as a comment on every pull request.
Real Experience: What Production Taught Us
Frameworks look clean on paper. Here is what actually happened when we applied this to a live AI support agent for a subscription-based client.
Issue 1: A model upgrade silently changed tool behaviour
We moved to a newer model version for better reasoning. Answers improved, but the agent became more “helpful” and started taking actions faster, including calling the cancel tool on ambiguous messages. Our old test suite only checked final text, so it passed.
Fix: we added trajectory assertions and a hard guardrail: destructive tools always require an explicit confirmation turn. We now pin model versions and treat any model upgrade like a dependency upgrade, run through the full eval suite first.
Issue 2: The judge was too generous
Our first LLM judge used the same model as the agent and a vague prompt (“Is this answer good?”). It scored almost everything 9/10. When we manually reviewed 100 cases, we disagreed with the judge on about one in four.
Fix: we switched to binary questions (grounded: yes/no, correct tool: yes/no), added rubric examples, and calibrated the judge against a hand-labelled set. Agreement with human reviewers rose to over 90%.
Issue 3: Flaky tests destroyed trust
Because outputs vary, some tests passed and failed randomly. Developers started ignoring red builds, which defeats the purpose of testing.
Fix: we stopped thinking in pass/fail per run and started thinking in pass rates. Critical cases must pass 3 out of 3 runs; normal cases need 2 out of 3. Temperature is set to 0 for evaluation runs where the product allows it.
Issue 4: Evaluation itself became expensive
Running 300 cases, three times each, with a judge call per case, on every PR, was costing more than the feature’s production traffic.
Fix: PRs now run a 60-case smoke set; the full suite runs nightly. We cache agent outputs when the prompt and model have not changed, and we use a smaller, cheaper model as the judge for simple rubric checks. Eval cost dropped by roughly 70%.
Measurable results
After eight weeks with the full pipeline in place:
- 62% fewer wrong tool calls, mostly from trajectory tests and the confirmation guardrail.
- 55% fewer ungrounded answers, after tightening retrieval and judging every answer against its sources.
- 83% fewer regressions reaching production, because risky prompt changes were blocked at the PR stage.
- 36% lower average cost per conversation. The eval data showed that shorter prompts and a smaller model for simple queries gave the same quality scores.
Biggest lesson: you cannot improve what you cannot measure. Once every prompt change came with a score, the team stopped arguing about opinions and started shipping improvements confidently.
Conclusion
Testing AI agents is not about forcing LLMs to be deterministic. It is about building a safety net that measures behaviour, catches regressions early and learns from real users. Start small: write 30 golden test cases from real conversations, add one LLM-as-judge scorer, and block merges when critical cases fail. Then grow the dataset every week from production feedback.
Key takeaway: treat prompts, models and retrieval data like code. Version them, test them and never ship a change without a score.
If your team is building an AI agent, chatbot or LLM-powered feature, or already has one in production that behaves unpredictably, we can help. At Logic Providers, we design and build AI agent and LLM integration solutions with evaluation pipelines built in from day one, so you get faster releases, lower API costs and AI features your customers can trust. Talk to our team to review your current setup.
.webp)
