How do I check that an AI-generated locator is stable?

Run the locator against the real page and assert it matches exactly one element, then run it again under conditions that change, such as a reload or different data. A locator built on something that changes between loads, like a generated id, fails that second run; a role, label or test id survives it.

This answer is one piece of Chapter 31: Reviewing AI-Written Tests.

Why Agent-Written Tests Pass Review
The Assertion That Isn't One
Tests That Pass for the Wrong Reason
Setup That Silently Does Nothing
Reading a Diff You Didn't Write
Accept, Fix, or Throw Away

An agent that was not given the real page guesses locators, and an agent that was given it often copies whatever attribute it saw first, including ids the app generates at runtime. Both read as plausible. Reading the line will not tell you which kind you have; running it will.

Two checks catch most of it. First, await expect(locator).toHaveCount(1) confirms the locator finds one element, not zero and not several. Second, change something that should not matter and run it again. Playwright's guidance is to prefer "user-facing attributes and explicit contracts", so a locator built on a label, a role or a data-testid should pass both runs.

The practice app has an input whose id comes from the seed: at ?seed=42 it is field-6605ab-1, and ?ids=random changes it on every load, the way many real apps do. In the example, an agent-style #field-6605ab-1 passes at seed 42 and finds nothing under ids=random. getByLabel('Report reference') passes in both.

ai-locator.spec.js(https://practice.learnplayforge.com)

import { test, expect } from '@playwright/test';

// The locator an agent copied from the DOM, and the one a reviewer swaps in.
const aiLocator = (page) => page.locator('#field-6605ab-1');
const labelLocator = (page) => page.getByLabel('Report reference');

for (const query of ['seed=42', 'seed=42&ids=random']) {
  test(`AI locator, ${query}`, async ({ page }) => {
    await page.goto(`/index.html?${query}`);
    await expect(aiLocator(page)).toHaveCount(1);
  });

  test(`label locator, ${query}`, async ({ page }) => {
    await page.goto(`/index.html?${query}`);
    await expect(labelLocator(page)).toHaveCount(1);
  });
}

Output

  1) › AI locator, seed=42&ids=random
    Error: expect(locator).toHaveCount(expected) failed
    Expected: 1
    Received: 0

  1 failed
  3 passed (9.2s)

Test yourself

An AI-written locator #field-6605ab-1 passes today. What is the quickest way to learn whether it is stable?

Practised in the full course: the drill Attached, hidden, and the id is gone.

See the chapter

AI can write the test now. The job is knowing whether it checks anything. That's what the course teaches, one chapter at a time.

163 lessons · 226 drills · start free

How do I check that an AI-generated locator is stable?

Related questions