AI test automation with Playwright: explore first, then replay
16 Nov 2026 · 5 min read · QEvin team
An AI agent can use a web app the way a person does. It reads the screen, clicks, types and notices what changed. That makes it good at exploring an app you have never seen. It does not make it a good test runner.
The better pattern has two steps. Let the agent explore the app once and write down what it learns. Then turn that knowledge into ordinary Playwright tests that run with no model at all. QEvin is built around this split: explore with computer use, replay with deterministic scripts.
Why not run a model in every test?
It is tempting to give an agent a goal such as "check that a guest cannot check out" and run it on every commit. This is a poor fit for a regression suite.
- It is slow. Each step waits for a model to look at the page and decide. A script just acts.
- It costs money on every run. A suite that runs on every pull request pays for model calls each time, even when nothing changed.
- It is not repeatable. Two runs can take different paths. When a run fails, you cannot tell whether the app broke or the agent chose differently.
- It is hard to review. A pass from an agent is a story about what it did. A pass from a script is a list of assertions you can read.
A regression test should do the same thing every time and fail for a reason you can name. That is what a script gives you.
Why record-and-replay is not enough
Recording a session is the old answer to the same problem. You click through the app once, the tool saves the steps, and you replay them later. It works until the page changes.
A recorder saves what it saw at that moment: a CSS path, an index in a list, a position on screen. It does not know why you clicked, so it cannot tell a harmless redesign from a real break. It also records only the one path you took. It does not tell you what else is in the app.
The explore step should record more than clicks. It should record what the app is made of.
Explore once, then replay
The two steps do different jobs, so they use different tools.
| Explore | Replay | |
|---|---|---|
| Runs | Once per area, then when the app changes | On every commit or release |
| Uses a model | Yes, to read the screen and decide | No |
| Output | Knowledge and candidate tests | Pass or fail with evidence |
| Cost | Paid once | Close to zero per run |
The agent does the part that needs judgment, which is finding out how the app works. The script does the part that needs discipline, which is checking it again and again.
What the explore step must capture
If the agent only hands you a finished test, you have to trust it. If it hands you what it saw, you can check it. QEvin is built to store four things for every area it explores.
- Pages. Each screen it reached, with the URL and the state it was in, such as logged out or cart not empty.
- Locators. How to find each control. Role and accessible name come first, because they follow what the user sees and survive layout changes.
- Transitions. What an action did. Clicking "Proceed to checkout" as a guest opened a dialog. That cause and effect is the raw material for an assertion.
- Evidence. A screenshot, the action log and the page state behind each fact, so a person can look at it.
Each fact also needs a status. Something the agent saw once is observed. Something it guessed is inferred. Something a script has confirmed is validated. Keeping these apart stops a guess from being shown as a fact.
What a replayed test looks like
Take a demo store. The agent explores it and finds that a guest who tries to check out is asked to log in. It writes the path as a Playwright test:
import { test, expect } from '@playwright/test';
test('guest is asked to log in at checkout', async ({ page }) => {
await page.goto('/shop');
await page.getByRole('link', { name: 'View product' }).first().click();
await page.getByRole('button', { name: 'Add to cart' }).click();
await page.getByRole('link', { name: 'View cart' }).click();
await page.getByRole('button', { name: 'Proceed to checkout' }).click();
await expect(page.getByRole('dialog', { name: 'Log in to check out' })).toBeVisible();
});
Nothing here needs a model. Every locator is getByRole with a visible name. The test ends with one assertion about behavior the agent observed. You can run it in CI, read it in a pull request and edit it by hand.
How to validate generated tests before you trust them
A generated test is a draft. Before it joins the suite, put it through the same checks you would apply to a colleague's test.
- Run it. It must pass against the app it was written from. A test that fails on day one is wrong.
- Run it again. Repeat the run to catch tests that pass only some of the time.
- Break it on purpose. Change the behavior it claims to check and confirm the test fails. A test that cannot fail is not a test.
- Check each locator. It should match one element, and the name should be one a user can see.
- Review the code. A person reads the diff and approves it. QEvin is designed so that nothing is published to your repository without that approval.
Record the outcome on the test: generated, reviewed or validated. When the app changes, mark the test stale instead of silently editing it.
What replay does not prove
A passing replay proves that the script ran and its assertions held in that environment. It does not prove that the test checks the right requirement. An agent that explored a bug can write a test that locks the bug in.
So keep two questions apart. Did the test run correctly? Is the test checking the right thing? The first is a machine's job. The second still needs someone who knows what the product should do.
Where QEvin fits
QEvin is an autonomous Quality Engineering computer for web apps. It is built to explore an app in a controlled browser, record pages, locators, transitions and evidence, and generate Playwright tests from that record. It is not released yet. You can read how the pieces fit on the home page.