Testing an agent as its developer
Market research and a hands-on lab, to check the assumptions in our design notes before we decide what to build.
In short
- Simulating the customer is already a feature: agent platforms build it in and frameworks give it away. Simulating the systems and people behind the conversation is not.
- We built a billing-support agent and tested it the way a team would, from mocks up to three parties, in 38 runs.
- With a clear policy and a world that tells the truth, the model kept every hard rule. Mocks couldn't tell right behavior from a double refund.
- What broke was level 2. With a customer, the agent and a manager in play, replies went to the wrong person, the agent never asked the person who knew, and a fact went unrelayed. Our own test harness broke seven ways, and each made a result lie.
- So the part worth building is parties with roles, a router for who hears what and when, and grading on world state that covers the process. A fake of a team's own tools is not it.
Where we started
The target is polished agent companies that need testing and playgrounds to build their agents. Individual developers, most of them vibe coders, are welcome too. Labs are not a near-term buyer. The first world is undecided, and the existing code is set aside.
The assumptions came from our design notes: teams have the eval loop but lack a world; the other side's behavior is the key need; level 2 is made of reactions over time; grade end state; mocks are not enough. We checked them two ways. We read how agent companies test today, and we built an agent and tried to test it ourselves.
How agent teams test today
| Layer | What exists | What it means for us |
|---|---|---|
| The simulated customer | Sierra, Decagon and Intercom Fin build it into their platforms. Google ADK and LangWatch Scenario give it away. In voice, Coval raised a $28M Series A in June 2026. | A feature, not a product. Selling "an LLM plays your user" walks into all of them. |
| The systems behind it | Practitioners go from mocks to recorded cassettes to a handful of live smoke tests. AWS Strands fakes tools with an LLM, with no story for repeatability. Twin vendors (Arga, Archal, Rystic, Jentic) are early, and Arga leans toward training. | Open. Nobody publicly describes testing the backends an agent acts on. |
| Runners and evals | Braintrust raised an $80M Series B in February 2026. LangSmith, Arize and LangWatch compete. OpenAI shuts its hosted Evals on November 30, 2026. | Consolidating. Plug into these and into pytest; don't build another runner. |
Sierra's post on simulations describes the simulated customer in detail, and says nothing about the systems behind the conversation.
The practitioners we found agree that mocks mislead. One calls a fully mocked suite "a test of a prompt, not of an agent", because it never sees wrong tool choices, timeouts, rate-limit cascades or partial failures. Others note that shared vendor sandboxes can't produce a 429 or a 500 on demand, and that live SaaS calls in tests bring throttling, flakiness and cost.
The likely buyer, by inference: a Series A–B vertical agent company. Its agents work inside its customers' systems, so it can't build fakes for every stack, and it is too small to build Sierra-scale simulation itself. The lab below tests the developer's side of that claim. The buyer's side needs real conversations.
The lab
We played the developer. The agent handles billing support for a fictional SaaS company. It runs on the Claude Agent SDK with Haiku 5.5, and its tools look up customers and charges, refund, cancel, and ask a manager in Slack for approval. Its policy is the kind a real team writes:
- Refunds are allowed only within 30 days.
- Anything over $100 needs approval.
- Never refund twice.
- Act only on the logged-in person's account.
- Never claim success after a failed call.
Around it we built a stateful fake of billing and Slack with fault hooks, LLM-played customers and a manager, an LLM judge, and pytest tests in seven rungs. When a test exposed a problem, we changed the agent and ran the tests again.
| The lab in numbers | Value |
|---|---|
| Agent code, with tools and policy | ~110 lines |
| Fake world, parties and tests | ~600 lines |
| Runs and agent turns | 38 and 85 |
| Model cost at list price, agent only | ~$17 |
| One agent turn | ~15 s, ~$0.30 |
| One three-party trial | 90–110 s, $1–3.60 |
| Time spent in the fake world | ~0 ms |
The test code came out to more than five times the agent's own code. With an AI coding assistant, each piece still took minutes. The fake for the agent's own tools was never the hard part.
What each rung caught
| Rung | Setup | Result | What it caught |
|---|---|---|---|
| 0 | Canned mock responses | 2/2 | On a timeout the agent called refund twice, and the usual assertion still passed. The canned charge list can't show that the first refund went through. Our own mock helper also had a bug. |
| 1 | Stateful fake, one message, end-state checks | 5/5 | Duplicate charge, the 30-day window, approval over $100, refusing someone else's account, cancellation at period end. Still 5/5 after the agent changed. |
| 2 | An LLM-played customer, several turns, three trials | 5/6 | A policy gap: with no exception path, the agent refused all escalation. Also two harness faults: the customer's stop marker cut off the agent's last action, and the judge contradicted itself. |
| 3 | A manager decides later; the event reaches the agent | 2/2 | Approved: refunded after the event. Denied: no refund, reason passed on. |
| 4 | A 429, then a timeout after the refund had committed | 5/5 | With a truthful world, the agent checked state after the timeout and did not retry. |
| 5 | An instruction hidden in a charge description | 2/2 | Ignored. Refunded the small charge and sent the large one for approval. |
| 6 | Customer, agent and manager; the manager asks first | 0/2, then 1/3 | The level-2 failures below. The first version of the agent failed outright. The second passed our checks 3/3 but only 1/3 on a stricter regrade. |
Where it broke: three parties
A customer asks for a $299 refund after an accidental upgrade. It needs a manager's approval. The manager will refund only if the new seats went unused, and only the customer knows whether they were used. This is the flow, and where it failed:
- Customer → agent"Can I get the $299 back?"
- Agent → manager, in SlackApproval request.
- Manager → agent"Were any of the new seats used?"
- Agent → customerNever asked, in 3 of 3 trials. The agent told the manager its tools can't see usage.
- Customer → agentVolunteered that two teammates used them. Our simulated customer leaked this hidden fact unprompted, 3 of 3.
- Agent → managerNot passed on in 1 of 3 trials. That approval stayed pending.
- Manager → agent → customerDenied, and the customer is told.
In the first version it was worse. The agent had one output channel, so its answer to the manager went to the customer, who replied that her name was Dana, not Lee.
In the agent
- Replies addressed to the wrong party
- No tool to answer in the approval thread, or to message the customer later
- Never asked the person who knew
- Didn't relay a fact once it had it
- Promised updates it had no way to send
In our harness
- A stop marker that ended the conversation before the agent acted
- The manager built from the customer's template
- A decision read from text: "APPROVE or DENY?" was recorded as approved
- A typo in the recorded status
- A persona that volunteered its hidden fact
- A judge whose reasoning and verdict disagreed
- Checks on the outcome alone, which missed the stuck approval
The model kept the rules. What failed was everything between the parties, including our own test code.
Our assumptions, checked
| Assumption | Evidence | Verdict |
|---|---|---|
| Teams have the eval loop and lack a world | A stateful fake of our own tools took minutes and was decisive where mocks misled. Nobody publicly addresses the systems behind the conversation. | HalfA minimal world for your own tools is cheap |
| The other side is the key need | A basic simulated customer took about 60 lines and is free in frameworks. Ours found gaps no fixture could, and also produced false results four ways. | HoldsThe bar is reliability, not existence |
| Level 2 is reactions over time and between parties | One injected event was trivial. Three parties broke the agent and broke our harness twice. | HoldsThe strongest evidence we have |
| Grade the world's end state, not text | State checks were stable in every trial. The judge contradicted itself. Reading decisions from text corrupted state. Outcome alone hid a stuck approval. | RefinedAlso grade the process: the decision, the relay |
| Agents are noisy, so report pass^k | The agent's outcomes were stable across trials. Most of the noise came from the simulated parties and the judge. | RefinedMake the people reproducible too |
| Mocks are not enough | The timeout test passes on mocks whatever the agent does, and our mock helper had its own bug. | Holds |
| Faults and injection are where agents break | Haiku 5.5 handled the 429, the ambiguous timeout, the injection and the impersonation in every trial. | WeakenedKeep as regression; not the first pain |
| Model cost, not environment speed, is what matters | About 15 s and $0.30 per turn, up to $3.60 per three-party trial; the world takes about 0 ms. | HoldsSuites run per release or nightly |
| The product lives in pytest | Everything we wanted was a fixture: the world, the parties, the checks. | HoldsOne developer, us |
| Reach through vendor MCP servers and SDKs matters | Untested. Our agent's tools were our own, so swapping in a fake was easy. | OpenAsk real teams first |
What we'd pay for, as this developer
- Parties with roles. A customer, a manager or a reviewer, each with hidden facts it reveals only when asked, decisions recorded as structured data, and record-and-replay so a failing trial reruns exactly.
- A router. The loop that decides who hears what, through which channel, when time passes and when the agent wakes up. We wrote it twice and got it wrong twice.
- Checks on world state that cover the process. Was a decision made, was the fact relayed, did the right person hear it, with pass^k across trials. Judges only for tone.
- Fault and injection hooks on the same world. Cheap once the world exists.
What we would not pay for: a fake of our own six tools, or an LLM that plays a customer on its own.
Questions for October 29
The first three decide the rest.
- Is the product parties, a router and state checks, and what is the smallest version a polished team would adopt this month?The lab says this is where both the agent and our harness failed. The market says nobody sells it. We have one developer's evidence, ours.
- Which workflow comes first?The lab suggests picking one with several parties, real waits and costly mistakes. Candidates: support with approvals, sales with a prospect and a manager, on-call with responders and stakeholders, pull requests with a reviewer and CI.
- Do target agents call vendor MCP servers and SDKs directly, or their own tool layer?If they call vendors directly, they need API-level twins of those vendors. If they own the layer, a fake of their own tools takes minutes, and our value is the parties and the router.
- Do we build the simulated customer or borrow it?LangWatch Scenario and Google ADK already do the basic version. Our effort may belong in roles, routing, replay and process checks.
- What does cost mean for the product?A three-party trial costs $1–3.60 in model calls, so suites run per release or nightly. Replaying the parties cuts the cost. What we charge for can't be compute minutes.
- What does a vibe coder get on day one?They can write a fake in minutes with a coding agent. A one-line fixture with ready-made parties and checks may be the hook. Or they may not need us.
- Who do we talk to first, what do we show them, and what answer would change the plan?The three-party failure is a sharper demo than the Slack-bot fix. It shows something their own tests would miss.
To ask real teams
- Do your agents call vendor MCP servers or SDKs directly, or your own tool layer?
- How many external systems, and how many other people (customers, approvers, teammates), does one workflow touch?
- What was your most expensive production failure: a broken rule, an external fault, or something lost between people or over time?
- How do you test multi-party or asynchronous flows today?
- What do you pay for evals now, in tools and in model spend per run?
Method and limits
This is one agent, one model and 38 runs. It is evidence, not a benchmark. The harness and the simulated parties are ours, and they had bugs, which is part of the finding. Costs are list prices as the SDK reported them. The runs went through a Claude Code subscription. The market section is desk research from public pages; vendor numbers are their own claims, and funding figures come from press and aggregators. The lab code and full notes are in ~/ancha-strategy/.
Sources
How teams test
- Sierra: Simulations, the secret behind every great agent (August 2025)
- Decagon: The future of AI agents is test-driven and Decagon simulations
- Intercom: Run simulations for Fin procedures
- Notion's agent evals, a third-party summary
- Tian Pan: mocked tool evals miss production failures (April 2026)
- Truto: testing MCP servers in CI without live APIs
- Anthropic: Demystifying evals for AI agents