Rethinking E2E Testing With Agents

Over the past few months, the bottleneck in my work has moved from writing code to verifying it. AI keeps making it easier to get a working implementation, but confirming that it does what I meant to deliver still takes a lot of my time. So I started automating testing, and the part that caught my attention most was end-to-end (E2E) testing.

Verifying what changes slowest

AI now writes almost all my code. It proposes most of the approaches too. My job is catching its mistakes and choosing between its options. Trying another implementation has become cheap enough that a module may be rewritten from scratch next month, and tests coupled to its implementation details get replaced along with it. Reviewing those tests closely is worth less to me than it used to be.

User stories change much more slowly. They describe what a user can do and what happens as a result, and they usually stay the same while the implementation underneath gets rewritten. Checking one usually means doing what the user would do, from getting the data into the right state, through the steps, to looking at what the system did at the end. That’s what an E2E test does, and it’s where more of my attention has gone. When I review an E2E test plan, I look at the happy path and the common error paths, then scan the concrete cases for anything that doesn’t make sense for what we’re delivering to the user.

Recently I saw a much more extreme take. Ansh Nanda shared the rules at the top of an AGENTS.md file: never write unit tests after writing the code, and “highly prefer E2E tests as the sole testing mechanism.” When a system has to be tested in isolation, the rules say to write all the ways it could fail first, then the code. I haven’t gone that way. Stable business rules are a natural fit for unit tests. I still want unit tests to cover as much of the code as possible, and integration tests to cover the essential cases without exhaustively testing every combination. AI now does almost all the writing and reviewing of these tests, so they take very little of my attention.

Why we used to write so few E2E tests

Our team’s process has every feature or bug fix tested on a feature branch server after implementation. Before I automated it, preparation alone could take half a day to a day: deciding what to test, building fixtures, getting the seed data into the right state. The database might already have a test customer, but before it can use the new feature, it needs a particular configuration. Understanding that configuration and setting it up is part of the test.

The process doesn’t stop at the branch server. After merging, I test again on the shared integration branch, where everyone’s changes come together, then again on the release-candidate server. The database on the release-candidate server is read-only, so any state change there has to go through the API as a simulated user action, which adds steps even to setup. Turning all of that into automated tests often took longer than the testing itself, so few of these checks ever became E2E tests.

Keeping E2E tests few is also what the usual advice says. Google’s 2015 post Just Say No to More End-to-End Tests, for example, argues that E2E tests give slow feedback, are flaky, and make failures hard to localize, so you should lean on unit tests, with a 70/20/10 split as a starting point. Martin Fowler’s explanation of the test pyramid is more careful about it: broad tests are usually more expensive, slower, and more brittle, but if those drawbacks go away, the balance can change.

As I read them, both arguments come down mostly to cost. In my case, most of that cost was human time: working out what to test, preparing the data, repeating the steps in each environment, checking the results. Slow feedback and hard-to-localize failures haven’t gone away. What I could change was how much of my own time each round took, without skipping the part where I check the results myself.

How I automated my E2E testing

When it comes to delivering software, I really like Simon Willison’s post Your job is to deliver code you have proven to work. Its argument is that delivering code means delivering proof that it works. I wanted the same from the agents. A final message saying everything passed wasn’t enough. I needed evidence I could check myself. So I built the automation in a way that shows, at every step, what it was trying to do and what happened, which also lets me step in at any point. And since the point of automating was to save my time, checking had to be fast too.

In practice, that proof comes from two things: a plan I review before any test code is written, and a log of every run. Before either reaches me, a subagent has already reviewed it. Both come out of a set of skills I wrote for planning, executing, and reviewing E2E tests, drawing on how I used to test by hand. The process starts by gathering context from the code, Git history, Jira, and our wiki, then checks with me on what I want and clarifies open questions. Then it writes a plan: for each case, what it tests, why, the user’s scenario, what data state it needs, and what counts as success. It also lists prerequisite checks, like verifying the seed data or depending on an earlier test, and cleanup so the test doesn’t interfere with later ones.

Writing the plan down for review has a side effect: it pins down which path the test must take. Slack’s agentic testing experiments show why that matters: in most runs, the agents reached the same goal by a different route. That flexibility helps during exploration, but if the bug is on one particular path, a run that reaches the goal by another route hasn’t tested it. The plan fixes the path, and execution is checked against it.

Once the plan is approved, the agent writes the test cases, and those same tests later run on my branch, the integration branch, and the release candidate. When they run, each test has a reviewer agent and a test agent: the reviewer breaks the work into small steps with preconditions, actions, assertions, and cleanup, and the test agent carries them out. The main session hands specific tests to subagents with separate contexts. Entry points stay as close to the user as the change allows. For a backend change with an unchanged interface, I generally test the API directly. When user interaction is involved, the agent operates the browser through Chrome DevTools MCP.

To show that the plan was followed, every run keeps a log of the raw API requests, responses, and assertions. At first I added it only for my own convenience, so I could see what happened, ask the agent to explain a step midway, or check it on my own. I didn’t expect the same log to work just as well for other agents. Now, before I look at a result, agents running on different LLMs cross-check the evidence, so by the time I read the log, the obvious problems are already flagged.

Being able to step in also changes what the agent should do when it gets stuck. When a step in the plan can’t proceed, I want it to stop and ask instead of improvising. Left alone, it has invented strange workarounds for problems a person could solve in seconds: with Chrome MCP already configured, it tried to generate credentials on its own and call the API with curl. With the plan and the logs in place, whoever steps in knows where to resume.

How the skills keep improving

Three kinds of documents keep changing as I use this setup. The codebase has its own documentation, organized by feature, which gives the agents context about the code. Each skill has its instructions (SKILL.md), and reference documents (references/*.md) the agent reads only when a task needs them.

The codebase documentation mostly builds up during the work. The agents rely on it for context, so they’re the ones who notice when it falls short: it doesn’t get them up to speed quickly, something in it is wrong, or it leaves out enough that they end up reading the code. When that happens, they update the documentation as part of the ticket and write it back to the repository.

The rest comes from stepping in. When an agent is missing something I know, or gets a plan or a decision wrong, the fix goes into the skill, so I don’t have to make the same correction on the next ticket. Where it goes depends on what it is. A short rule about how to test goes into SKILL.md. Missing context, or a fix that takes several steps, goes into the reference documents. For example, an agent once tried to rebuild the database locally because it didn’t know it could connect to the one on my feature branch server, which already had the seed data it needed. That went into the reference documents. I found a fitting name for this in Every’s guide: compound engineering.

The half day of preparation I described earlier, repeated on three servers, now mostly goes to agents. By my rough estimate, the time I spend on E2E testing has dropped to about a tenth. A single ticket isn’t much faster, but several can go through at once, each on its own branch server, while I do other work. What’s left for me is reviewing the plan and reading the log of the final run, since the agents check the logs of the runs before it. It’s a lot like listening to another engineer explain how they tested something. I’m checking whether the approach is sound and whether the evidence supports the conclusion.

A few things that make the results easier to trust

Reading the log of the final run only helps if a pass in it means something. For a bug fix, each E2E test case states both what should happen with the fix and how it should fail without it. The test has to fail that way on the integration branch and pass on the fix branch, since a pass alone doesn’t show it could see the bug. We used to skip this because it meant preparing everything twice, but now the same test cases carry over.

For operations that change stored data, the tests look at the database. The API response only tells me what the system says happened. The database shows what did happen, and the two can disagree: the API itself may be wrong, or a cache that wasn’t cleared can make the response look right when the data isn’t. So each test checks that the database reached the state described in the plan. This reminds me of an example in Anthropic’s guide to agent evals: a flight-booking agent might say “Your flight has been booked,” but the outcome is whether a reservation exists in the database.

Also, checking the end state only means something if the data started where the plan expected, so each test looks after the data it touches. Before a case runs, it checks that the data it needs is in the expected state and restores it if an earlier test changed it. Afterward, it cleans up what it created, so one test’s leftovers can’t make the next one fail.

Deciding what to keep

Each ticket now produces far more test cases than before, and only some of them end up in the regression suite. To decide which ones, I go by a few rules of thumb:

• Could the problem come back, and how much would it hurt if it did? Code that changes often is more likely to break again, and a rare problem can still be worth a test if the damage would be serious.
• Would a smaller test have caught the original bug? If it came from components working together, a unit test would leave out exactly the part that broke, so it stays E2E. Otherwise a unit or integration test is often enough.
• Is it worth the cost of keeping it around? Data that’s hard to set up reliably, interference with other tests, or failures that are hard to track down all count against it.

The tests that do stay run as plain test code, and the agents are only involved while a ticket is being tested. I’ve only estimated the difference roughly and never measured it, but Slack’s experiments put numbers on it: an agent-driven run cost roughly $15 to $30 and took 5 to 11 minutes depending on the setup, while the generated scripts ran in about 32 and 45 seconds. Slack’s takeaway was to keep deterministic tests in CI and use agents for exploration and debugging.

How many tests to keep is part of an old question: how much testing is enough. I don’t aim for a coverage number. Fowler’s Test Coverage treats coverage as a way to find untested code, not as a measure of test quality. What it looks at instead is whether bugs escape to production and whether people can change the code with confidence. Google’s How Much Testing Is Enough? answers that it depends on the type of software, its purpose, and its audience. For me, it comes down to whether I’m confident in what we’re delivering, and I still get that confidence from knowing the system and from experience, as I did before AI.

Changes that cross systems

Within a single system, automated E2E testing is now routine for me. One direction I’m interested in next is changes that cross systems. A change can look fine in its own service and still break something a downstream service depends on. An E2E test across systems can catch that, though a contract test between the two services is often enough. Either way, the agents need to know how the systems work together and where each one’s responsibilities end, and that knowledge has to stay accurate as the code in every system changes. Within one system, that knowledge lives in the codebase documentation and the skills’ reference documents, which build up during the work and get corrected when I step in. For knowledge that spans systems, one idea is to have the agents work out a way to make sure it’s correct and keep it current.

I haven’t tried any of this across systems yet. So far I’ve been reading about how other teams handle parts of it:

• Uber’s BITS indexes the traces of test runs, linking each test to the services and endpoints it touched, and a more recent Uber post infers dependencies between services, and whether each one is hard or soft, from request-level failure data in production. Both show what runs between systems, though not the paths nothing has exercised or why the systems were built to work that way.
• Nubank replaced its end-to-end test suite with contract tests built from the schemas its services already declared, after finding that schema violations were the most frequent kind of bug the suite caught. The post notes that contract tests catch structural incompatibilities but aren’t good at testing behavior, and Nubank kept acceptance tests for critical business flows.
• Shopify’s Under the River describes River, an AI agent that works in the company’s Slack. Shopify keeps everything it ships in one repository, which also holds skills, runbooks, and AGENTS.md files, and notes that agents can’t see across a fragmented one. When someone solves a problem with River, they leave something behind, such as a skill update or an AGENTS.md diff.
• Michael Nygard’s Documenting Architecture Decisions, from 2011, is about recording why a system was built the way it is: “One of the hardest things to track during the life of a project is the motivation behind certain decisions.” The records live in the project repository, and a reversed decision is kept but marked as superseded.

If I get to it, I’ll write about how it goes.