What is a flaky test, and how do I fix one in Playwright?

A flaky test is one that passes on some runs and fails on others with no change to the code; Playwright's report labels a test flaky when it failed first and then passed on retry. You fix one by reproducing it with --repeat-each, finding what it depends on that changes between runs, and removing that dependency.

This answer is one piece of Chapter 30: Flakiness at Scale.

Flaky, or Simply Broken
Reproducing Flake on Purpose
Reading a Trace for the Real Cause
Shared State and Test Pollution
A Suite the Team Has Stopped Trusting

Playwright's retries page sorts results into three groups: passed, flaky ("failed on the first run, but passed when retried") and failed. A retry that turns red into yellow does not fix anything. It hides it.

The first step is to make the flake happen on demand. npx playwright test --repeat-each=10 runs each test ten times. A test that fails every time is not flaky, it is broken; a test that fails some of the time is depending on something that changes between runs. Common causes are timing, shared data and randomness.

This example uses randomness. On the practice app, the catalogue's Recommended strip is shuffled on every load when the URL has no ?seed=. The first test leaves the seed out on purpose and assumes the kettle is in the first slot.

Run ten times each, the first test passed once and failed nine times; a second run of ten gave a different count, with a different product, such as the grinder or the dripper, in that slot each time it failed. The second test passed every run. It pins ?seed=42 and checks that the kettle is in the list, not where. Either change alone helps; together the test no longer depends on luck.

flaky.spec.js(https://practice.learnplayforge.com)

import { test, expect } from '@playwright/test';

// Flaky: no ?seed=, so the Recommended strip is shuffled on every load,
// and the test assumes a position.
test('flaky: assumes the first slot', async ({ page }) => {
  await page.goto('/catalog.html?mock=static&autologin=1');
  const items = page.getByTestId('recommended-item');
  await expect(items.first()).toContainText('Aurora Pour-Over Kettle');
});

// Stable: seed pinned, and the check is about content, not position.
test('stable: checks membership, not order', async ({ page }) => {
  await page.goto('/catalog.html?mock=static&seed=42&autologin=1');
  const list = page.getByTestId('recommended-list');
  await expect(list.getByTestId('recommended-item')).toHaveCount(6);
  await expect(list.getByText('Aurora Pour-Over Kettle', { exact: true })).toBeVisible();
});

Output

$ npx playwright test flaky.spec.js --repeat-each=10

    Locator: getByTestId('recommended-item').first()
    Expected substring: "Aurora Pour-Over Kettle"
    Received string:    "Northwind Knock Box"

  9 failed
  11 passed (27.9s)

Test yourself

A test fails on all ten runs of --repeat-each=10. Is it flaky?

Practised in the full course: the drill The unseeded assumption.

See the chapter

AI can write the test now. The job is knowing whether it checks anything. That's what the course teaches, one chapter at a time.

163 lessons · 226 drills · start free

What is a flaky test, and how do I fix one in Playwright?

Related questions