(This lesson is part of Act V, PlayForge's capstone on AI-era automation. It is available to the public as a free sample.)

An agent writing Playwright gets the shape of a test right almost every time. A test() block. A locator built with getByRole. An await on every line that needs one. A variable called loginButton that really does hold a login button. That shape was learned from an enormous number of correct tests, and it comes out clean.

What the shape does not carry is whether the test can fail. That is a property of the run: does this line block until the app settles, does this comparison retry, does this assertion have anything to compare against. The agent has no run. It has text that looks like test code.

So the defect that reaches your pull request rarely looks like gibberish. It reads as an accurate English description of the thing you wanted checked, and it verifies nothing.

The case this course can prove

This course was itself largely written by an AI agent, then audited lesson by lesson against playwright.dev. The locator chapter — six lessons, the chapter the syllabus builds toward — contained sixteen occurrences of this line shape:

await page.getByRole('heading', { name: 'Dashboard' }).isVisible();

isVisible() returns a boolean. That line awaits it, discards it, and moves on. The API reference is blunt about the timing: "locator.isVisible() does not wait for the element to become visible and returns immediately." Delete the heading from the page entirely and the line still passes.

The second number is the one worth sitting with. All sixteen were in the code. Zero were in the prose. Every paragraph in that chapter explained checking an element correctly, named expect, and described retrying. A reviewer reading the lesson would have approved it without hesitation. The wrongness lived entirely in the part that gets skimmed.

That generalises well past this course. Agent-written prose describes the correct behaviour while agent-written code does something else, because the two were generated separately and neither was checked against the other. The PR description is a claim about intent. It is never evidence about the diff.

Six patterns worth having names for

Each of these appeared in real generated Playwright in this repository:

The discarded return value. isVisible(), count(), textContent(), getAttribute() — called, awaited, thrown away. Some of these return instantly and some wait for the element to appear, and none of them compares the value to anything.
The one-shot comparison. expect(someValue).toBe(...) where the value was read once. It cannot retry, so it is a race with the app's rendering.
The negative assertion that can't fail. toBeHidden() or not.toBeVisible() pointed at a locator that matches no element. Green forever, including when the feature is broken.
Setup that never takes effect. A page.route() with no navigation after it. A config option written at the wrong level and silently ignored.
The right words on the wrong element. getByText('Saved') matching a hidden toast template instead of the live one.
Timing papered over with a sleep. waitForTimeout(3000) in place of an assertion that would have waited exactly as long as needed.

Notice how small each one is. isVisible() against toBeVisible() is a two-word difference that changes a line from a check into a no-op. That is why reviewing generated tests by reading them for sense does not work — the sense is intact. The next three lessons take the biggest classes apart one at a time.

javascriptruns against a real browser

// A generated spec as submitted, from the prompt:
//   "Write a Playwright test that logs in and checks the dashboard loaded."
// It reads correctly end to end. Three lines check nothing.

import { test, expect } from '@playwright/test';

test('user logs in and sees their dashboard', async ({ page }) => {
  await page.goto('https://shop.example.com/login');

  await page.getByLabel('Email').fill('priya@example.com');
  await page.getByLabel('Password').fill('correct-horse');
  await page.getByRole('button', { name: 'Sign in' }).click();

  // REVIEW 1: boolean discarded. Returns immediately, never waits,
  // passes whether or not the heading is on the page.
  await page.getByRole('heading', { name: 'Dashboard' }).isVisible();

  // REVIEW 2: page.title() returns a Promise. This compares a Promise
  // object to a string and fails at once with `Received: Promise {}`.
  // Loud, so it is the easy one to catch.
  expect(page.title()).toBe('Dashboard');

  // REVIEW 3: nothing renders in a fixed 3 seconds on purpose. This is
  // a guess standing in for an assertion, and it is the slowest line here.
  await page.waitForTimeout(3000);

  // REVIEW 4: the only real check in the file, and it is the weakest
  // claim the test could make.
  await expect(page).toHaveURL(/dashboard/);
});

31. Reviewing AI-Written Tests

Why Agent-Written Tests Pass Review

The case this course can prove

Six patterns worth having names for

5 more checkpoints in this chapter

Practice