Why AI-generated tests need evidence
16 Nov 2026 · 6 min read · QEvin team
You can trust an AI-generated test only as far as you can check it. A green result is not enough. A generated test earns trust when every step and every assertion points to evidence you can open: a screenshot, a DOM snapshot, the action that produced it, and the app version it ran against. If a tool cannot show that, treat its tests as drafts.
Can a generated test pass and prove nothing?
Yes, and it happens in a few predictable ways.
- The assertion is too weak. The test asserts something that is always true, so it can never fail.
- The assertion repeats the model's assumption. The model guessed that a button saves the profile, then wrote a test that expects the guess. Nothing in the app confirmed it.
- The locator matches the wrong element. A text match hits a heading instead of a button, or the first of three similar links. The click lands somewhere else and the test still moves on.
- The test never reaches the step it claims. A redirect, a cookie banner or a login wall stops the flow early. The final check runs on the wrong page.
Here is a test that looks fine in a pull request and checks almost nothing:
import { test, expect } from '@playwright/test';
test('guest is asked to log in at checkout', async ({ page }) => {
await page.goto('/demo/cart');
await page.getByRole('button', { name: 'Proceed to checkout' }).click();
// Always true on any page that renders.
await expect(page.locator('body')).toBeVisible();
});
It passes if checkout works. It also passes if checkout is broken, if the login prompt is gone, and if the click opened an error page. The fix is an assertion that can fail for the right reason:
await expect(
page.getByRole('dialog', { name: 'Log in to check out' })
).toBeVisible();
A reviewer can spot this once. A tool that writes hundreds of tests needs a system that spots it every time.
Why is the model's explanation not proof?
A model writes fluent text. "I clicked the checkout button and saw the login prompt" reads the same whether it happened or not. The narrative is a claim, not a record.
QEvin is built on a simple split. The model proposes an action. A tool gateway checks policy and allows or blocks it. An executor runs it in a real browser. A verifier records what actually happened. The record comes from the browser, not from the model's description of it.
What evidence should a generated test carry?
Each fact behind a test should answer five questions.
- What did the tool see? A screenshot of the page at that moment.
- What was on the page? A DOM or accessibility snapshot, and how many elements the locator matched. The count should be exactly one.
- Which step produced it? The action in the run, so you can replay it and see the same thing.
- When, and on which build? A timestamp and the app version. A fact with no version cannot be checked against a new release.
- Has it been repeated? A replay in a clean browser, with the result recorded.
A small JSON shape shows what a recorded fact can look like. The values here are illustrative.
{
"claim": "Guests are asked to log in at checkout",
"status": "validated",
"evidence": {
"screenshot": "cart-after-click.jpg",
"domSnapshot": "cart-after-click.html",
"locator": "getByRole('button', { name: 'Proceed to checkout' })",
"locatorMatches": 1
},
"provenance": {
"step": 8,
"observedAt": "2026-01-01T10:00:00Z",
"appVersion": "2.4.0-staging"
},
"replay": { "runs": 3, "passed": 3 }
}
What does a status on every fact mean?
Evidence has strengths. A tool should say which one it holds, in plain words, on every fact. QEvin is built to use six statuses.
| Status | Meaning | Example |
|---|---|---|
| Observed | Directly seen in a recorded browser state. | A button labelled Save exists on the page. |
| Inferred | A plausible model guess, not yet verified. | Save probably stores the profile. |
| Validated | Confirmed by a repeatable interaction or replay. | Save showed a confirmation and the data persisted. |
| Human-confirmed | Reviewed and accepted by a person. | A tester confirms this route is the billing overview. |
| Stale | Was valid, but the app changed or the evidence is old. | A locator no longer resolves on the latest build. |
| Conflicted | Observations disagree. | A route behaves differently by role. |
Two rules matter most.
Inferred is never shown as fact. A guess can be useful. It can become a test idea. It must stay labelled as a guess until something observes or validates it.
Inconclusive is never a pass. Sometimes a run cannot finish its checks: the environment fell over, the page did not load, the data was missing. That result is "inconclusive". It is not green. Counting it as a pass is how a release gate ends up trusting nothing.
How do I review an AI-generated test? A checklist
Use this on any AI testing tool.
- Read the assertion. Could it fail if the feature broke? If it checks only that a page rendered, reject it.
- Ask where the expected value came from. Observed behavior, a requirement, or the model's assumption? Assumptions need a person to confirm them.
- Check each locator. Does it match exactly one element? Does it use a role and an accessible name instead of a generated class or ID?
- Open the screenshot for the final step. Is it the page the test claims to reach?
- Look at the path. Did anything redirect, block or skip on the way?
- Find the version. Which build was this recorded on, and is it still current?
- Run it in a clean browser, more than once. One pass is a coincidence.
- Break the app on purpose. Change the expected text or hide the button. Does the test fail?
- Check how failures are reported. Does the tool separate a product bug, a test defect, an environment problem and an inconclusive run?
How is QEvin built around this?
QEvin is built to explore a web app in a controlled browser and keep what it learns in a knowledge graph. Every node and edge is meant to carry a status, provenance, a confidence value and the version it was seen on. Tests are generated from that graph, not from a screenshot or a free-form prompt alone. Each step links to a page and an observed locator. Each expected outcome links to observed behavior, a requirement, or an assumption that is stated as one.
QEvin will replay each generated script before showing it to you, and it will report a failed or inconclusive replay as exactly that. It is also built to ask a person before it does anything destructive or publishes code. The rules live in code, not in a prompt.
For how those scripts are produced and run, read AI test automation with Playwright: explore first, then replay. For the structure that holds the facts, read what a knowledge graph of your app gives a QA team. The product overview is on the QEvin home page.