What should I look for when reviewing a Playwright test in a pull request?

Start with the assertions: for each test, ask what would have to break in the app for it to go red, then check the names, waits, locators and data against the diff. The fastest proof is to break the behaviour on purpose and see whether the test notices.

This answer is one piece of Chapter 31: Reviewing AI-Written Tests.

Why Agent-Written Tests Pass Review
The Assertion That Isn't One
Tests That Pass for the Wrong Reason
Setup That Silently Does Nothing
Reading a Diff You Didn't Write
Accept, Fix, or Throw Away

A pull request gives you the diff, not the reasoning, so review in a fixed order.

Assertions first. Find each await expect( line. A test with forty lines of steps and one weak check at the end is a script, not a test.
Names against bodies. A test called adds the grinder should check something that is only true once the grinder is added.
Waits. Search the diff for waitForTimeout and networkidle; a web-first assertion should do the waiting.
Locators. Confirm each one exists in the real page and is spelled as the test spells it, especially under .not and toBeHidden().
Data. Look for values copied from one test run, and tests that depend on another test's leftovers.

Then break it. In the example, the reviewer comments out the Add to cart click in both versions. The PR's toBeVisible() on the cart badge still passes, because the badge is visible at 0. The reviewed toHaveText('1') fails with Received: "0", which proves it was checking the add all along.

pr-review.spec.js(https://practice.learnplayforge.com)

import { test, expect } from '@playwright/test';

// The reviewer's check: comment out the action the test name promises,
// then see which assertion notices.

test('PR version: adds the grinder (weak assertion)', async ({ page }) => {
  await page.goto('/catalog.html?mock=static&seed=42&autologin=1');
  // await page.locator('article[data-sku="MRD-1003"]')
  //   .getByRole('button', { name: 'Add to cart' }).click();
  await expect(page.getByTestId('cart-count')).toBeVisible();
});

test('Reviewed version: adds the grinder (strong assertion)', async ({ page }) => {
  await page.goto('/catalog.html?mock=static&seed=42&autologin=1');
  // await page.locator('article[data-sku="MRD-1003"]')
  //   .getByRole('button', { name: 'Add to cart' }).click();
  await expect(page.getByTestId('cart-count')).toHaveText('1');
});

Output

  1) › Reviewed version: adds the grinder (strong assertion)
    Error: expect(locator).toHaveText(expected) failed
    Expected: "1"
    Received: "0"

  1 failed
  1 passed (7.2s)

Test yourself

You comment out the click a test is named after, and the test still passes. What does that tell you?

Practised in the full course: the drill The typo that passes.

See the chapter

AI can write the test now. The job is knowing whether it checks anything. That's what the course teaches, one chapter at a time.

163 lessons · 226 drills · start free

What should I look for when reviewing a Playwright test in a pull request?

Related questions