Green After a Retry: Diagnose Playwright’s Hidden Test Failures

A Playwright job can finish green even though the same logical test failed minutes earlier. That is not automatically a false alarm, and it is not automatically a product defect. It is an evidence problem: the first attempt, the retry, and the system state between them may not be equivalent.

This playbook is for QA automation engineers, frontend test maintainers, CI owners, and engineering leads using the Node.js Playwright Test runner. The examples are scoped to a pinned @playwright/[email protected], released by Playwright on September 4, 2026; verify your own installed version before copying version-sensitive configuration. Playwright’s current documentation says retries are disabled by default, classifies eventual-pass tests as flaky, and replaces the failed worker process and its browser before continuing or retrying.

The documented runner behavior is only one layer. Browser contexts isolate cookies, local storage, and other browser-profile state, but that contract does not erase arbitrary database rows, queue messages, test accounts, files, or third-party sandbox state.

The operating model below is therefore deliberately stricter than “retry and see.” Preserve the first failure, record what changed, reproduce the suspected cause, and then make one of four accountable decisions: fix, quarantine, hold the release, or declare the evidence insufficient.

Read the Final Status Without Losing the First Attempt

Start with the timeline, not the final badge. With one configured retry, a logical test can produce attempt 0 = failed and attempt 1 = passed. Playwright reports that logical test as flaky, not as a first-run pass. A test that passes on attempt 0 is passed; one that fails its initial run and all retries is failed. Retries are disabled unless you enable them.

That distinction matters because three layers can disagree without contradiction. The CI job reports whether the command met its exit policy. The logical test reports its Playwright outcome. Each attempt reports the conditions and evidence of one execution. A green job can therefore coexist with a flaky logical test unless the configured gate makes flaky outcomes fail the command. Playwright provides failOnFlakyTests for that purpose, but it is an opt-in configuration control, not the default.

Do not compress “fail → pass” into “passed.” The retry answers an execution question: can this scenario succeed on another attempt? It does not, by itself, answer a confidence question: did the requirement hold under the conditions that caused the first failure?

This is the same review boundary that matters when automation can modify tests: a final successful execution is not proof that the test still means what it was intended to mean. Refonte Learning’s related discussion of Playwright AI-agent review boundaries makes that distinction in an agent-repair context; here, keep the same discipline for ordinary runner retries.

First triage checklist (proposed operating model): record the logical test outcome; preserve attempt 0 error and artifacts; record every retry index; compare worker, browser, data, and backend identifiers; and refuse to call the issue fixed until the same requirement passes without a retry-only change in preconditions.

Inventory the Runner, Suite and CI Execution Context

Two runs that display the same test title can still be materially different. Before diagnosing a flaky outcome, create a configuration fingerprint that lets another engineer reproduce the execution boundary instead of guessing it.

For this article, the version assumption is Node.js Playwright Test pinned to @playwright/[email protected]. Playwright’s official GitHub release page identifies v1.63.0 as a September 4, 2026 release. A team using 1.51, 1.52, 1.62, or 1.63 does not have an identical runner feature set, so the lockfile and actually installed package version belong in the evidence.

Capture the runner and application separately. The test runner fingerprint should include commit SHA, package-manager lockfile digest, @playwright/test version, browser binaries, Node.js version, operating system or container image, project name, workers, retries, shard, test filters, and CI attempt identifier. The application fingerprint should include server build, deployed commit or image digest, feature-flag set, database schema or data-seed version, and test-service version.

A whole CI job rerun is not the same event as testInfo.retry === 1. A platform rerun may recreate containers, re-seed databases, obtain different credentials, use a new CI attempt number, or deploy a different application build. Those effects depend on the CI and environment design, so treat job-level reruns as separate runs with their own provenance rather than assuming they reproduce a runner retry.

The minimum durable record is a run ID plus a configuration fingerprint stored with the reporter output. Prefer machine-readable metadata over console-only text. If you cannot later prove which runner, browser, application build, and test-data contract produced the failure, classify the investigation as evidence-limited rather than filling the gap with assumptions.

Inventory checklist (proposed local policy): fail the diagnostic intake if the Playwright version is unknown; preserve the lockfile; capture the resolved project and retry count; record browser versions; record the server build separately; record shard and worker count; and include a unique CI run/attempt identifier in every artifact bundle.

Follow the Worker Replacement Timeline

A Playwright retry is not execution resuming at the failed assertion. The official retries guide says tests run in independent worker operating-system processes, each worker starts its own browser, and when a test fails Playwright discards the entire worker process and browser and starts a new worker. With retries enabled, the replacement worker starts by retrying the failed test, then continues. beforeAll and afterAll participate in that lifecycle.

A compact attempt trace for one failing test looks like this:

workerIndex=3, parallelIndex=1
  beforeAll
  beforeEach
  test attempt retry=0  -> FAIL
  afterEach / afterAll as runner can execute them
worker exits; browser from that worker is discarded

workerIndex=7, parallelIndex=1
  beforeAll again
  beforeEach again
  test attempt retry=1  -> PASS
  afterEach / afterAll

The exact hook path depends on where the failure occurs, but the important diagnostic fact is the process boundary. Playwright’s parallelism documentation likewise says workers are shut down after a test failure so following tests get a pristine worker environment.

A runner-controlled restart can clean in-memory globals and browser state while leaving external writes untouched. If attempt 0 created an order, uploaded an object, consumed a one-time token, or queued work before it failed, attempt 1 can begin in a different business state even though it has a fresh browser. That external-state conclusion is a harness boundary, not a promise made by Playwright’s worker restart.

What beforeAll and beforeEach Can Repeat

Playwright documents that TestInfo is available in tests and hooks and that testInfo.retry is 0 for the first run, 1 for the first retry, and so on. That makes repeated initialization observable, but observability is not correctness.

If beforeAll provisions [email protected], the replacement worker can encounter setup that provisions the same account again. If beforeEach seeds order-123, the retry can encounter a record created or partially created by attempt 0. Design setup to be idempotent where repeated execution is truly equivalent, or use unique ownership so each attempt can determine which resources it may mutate.

Hook checklist (proposed model): record hook start and end; include retry, worker, namespace, and resource IDs; make expected repetition explicit; treat a successful second initialization as a changed condition until proven equivalent; and test the setup path under interruption, not only under clean completion.

Why Serial Groups Need Their Own Analysis

Serial groups change the retry unit. Playwright documents that when one test in a serial group fails, subsequent tests in that group are skipped, and with retries enabled the tests in the group are retried together.

That means an eventual pass may include earlier group members running twice and later members running only on the retry. Do not read “one flaky test” as “only one scenario changed state.” The group may have replayed setup-producing steps.

Serial diagnostic: record which members executed, skipped, and re-executed on every group attempt; map side effects by member; reproduce the failing member independently where possible; and only then decide whether restructuring improves isolation. Fewer visible red results are not evidence of better requirement coverage.

Separate Browser Isolation from Shared Application State

Playwright’s isolation model is strong inside the browser boundary: each test gets an isolated BrowserContext, with separate cookies, local storage, session storage, and related profile state. The isolation guide describes contexts as incognito-like profiles and says Playwright Test creates an isolated context for each test.

Do not extend that statement beyond what it says. A browser context is not a transaction around your database, object store, queue, account directory, email sandbox, payment simulator, or external test API. Those boundaries belong to the application, harness, or disposable test environment.

This boundary explains a common retry illusion. Attempt 0 creates server record order-X, then fails before teardown. Attempt 1 gets a clean browser context, so its cookies look fresh. The server can still have order-X. A “create order” action may now fail because that record exists, or pass only because retry-specific cleanup removed it. The fresh browser context did not, by itself, restore the business precondition.

Use tenant-scoped or run-scoped synthetic data instead of a production-like shared identifier. The discipline is foundational when moving from manual QA to automation: automated concurrency forces explicit ownership rules that a human tester can otherwise carry implicitly.

Isolation checklist (proposed model): list every mutable state store touched by the test; mark which boundary the runner controls; assign an owner for everything external; use only explicitly disposable test services; never run destructive cleanup against production accounts; and treat “fresh browser” and “fresh business state” as separate assertions.

Make Resource Ownership Survive Parallel Runs and Retries

Resource identity must survive the diagnostic questions you will ask after a crash. workerIndex is useful evidence, but it is not a durable business key. Playwright documents that a restarted worker receives a new workerIndex, while its parallelIndex remains the same.

A safer namespace is harness-owned. One practical pattern is:

<run-id>/<project>/<case-key>/<resource-policy>/<resource-key>

run-id prevents cross-run collision. project separates browser or device projects when their backend data must not collide. case-key is an explicit stable identifier owned by the suite, not a worker number. resource-policy states whether the resource persists across retries for forensic comparison or is recreated per attempt. resource-key identifies the business object.

Apply collision and length rules before you reach CI. Hash long test identifiers where an external system has strict key limits, while preserving the original identifier in ledger metadata. Use a character set accepted by every downstream system. Make cleanup select by ownership metadata or namespace prefix, never by a broad business predicate such as “delete all test orders.”

Playwright’s testInfo.testId can help correlate the runtime case; its API documents testId as the test identifier corresponding to the reporter API. I would still keep an explicit suite-owned case key for external resources because test source structure and names can change independently of the external data contract.

Record Both Worker Indexes

Playwright exposes both testInfo.workerIndex and testInfo.parallelIndex. The parallelism documentation says the worker index is unique to the worker; when a failed worker is restarted, the replacement receives a new workerIndex while retaining the same parallelIndex.

Put both in the attempt ledger. They tell you whether execution crossed a worker boundary and whether it occupied the same parallel slot. They do not prove exclusive ownership of an external account, database row, object, or queue message.

Ledger fields: runId, project, case key, testInfo.testId, retry, workerIndex, parallelIndex, resource namespace, server build, data version, request or correlation IDs, outcome, and artifact references.

Clean Only Resources Owned by the Attempt

Cleanup should be a scoped ownership operation. Attach metadata such as test_run_id, case_key, attempt, and lease_id to synthetic backend records where the disposable test service permits it. Cleanup queries should select exactly those records that the run or attempt owns.

Teardown also needs a crash strategy. ,c 39says test-scoped fixtures are torn down after each test, while worker-scoped fixtures are torn down when the worker process executing the tests is torn down. That runner lifecycle cannot guarantee that an external cleanup request succeeds if the external system is unavailable, the process is force-killed outside normal handling, or the cleanup itself errors.

Recovery rule (proposed local policy): use normal scoped teardown first; retain ownership metadata after a crash; run a separate janitor that deletes only expired leases from the disposable test environment; and exercise an interrupted-teardown test periodically. Never make “delete everything in the test tenant after any failure” the default recovery action.

Capture the Original Failure Before Optimizing Artifact Cost

Artifact policy should answer a question, not merely minimize storage. Playwright recommends trace: 'on-first-retry' for CI and documents that this mode records when a failed test is retried for the first time. It therefore captures retry execution, not the original attempt that already failed.

That recommendation is economical when the objective is to obtain a trace on a retry. It is insufficient when your diagnostic question is, “What happened on attempt 0 before a passing retry changed the state?” The same trace guide documents other modes including on, on-all-retries, and retain-on-failure; retain-on-failure records a trace for each test and removes it from successful test runs.

The trace guide and this diagnostic recommendation optimize different things; they are not contradictory. on-first-retry controls routine CI cost. Preserving attempt 0 optimizes causal investigation when a retry might pass under changed conditions.

Do not rely on the trace alone. Correlate application logs, server request IDs, queue or message IDs, database observations, and test-data ownership with the attempt ledger. Playwright’s Trace Viewer exposes execution information including actions, DOM snapshots, screenshots, console information, and network activity, but an important backend mutation may only be reconstructable from service-side evidence.

Artifacts can contain application text, request and response details, identifiers, and potentially sensitive data. Apply your organization’s access, retention, and redaction controls rather than treating traces as harmless debugging files. Refonte Learning’s article on visual regression evidence and baseline review addresses a different evidence class; keep visual-baseline approval separate from retry causality.

Artifact checklist (proposed policy): for a diagnostic cohort, preserve attempt-0 trace or equivalent failure evidence; attach the first error; record request IDs; snapshot the relevant owned backend record; keep retry artifacts separately; apply appropriate access controls; record missing artifacts as an explicit evidence gap; and only reduce retention after you know which fields actually falsify competing hypotheses.

Work Through a Synthetic Stale-Record Failure

The following example is synthetic. It was not executed for this article, and the ledger values are expected results, not measured output. Use only an explicitly disposable test service; do not point cleanup code at production accounts.

Suppose the requirement is: “A tenant with no existing order under this test-owned key can create one order, and the API or UI returns its identifier.” A previous interrupted test has left a synthetic row. A bad test reuses one fixed key and clears it only when testInfo.retry > 0.

// Assumption: Node.js Playwright Test @playwright/[email protected].
// Synthetic example only; api.* represents a disposable test-service client.
import { test, expect } from '@playwright/test';

test('creates one order', async ({ page }, testInfo) => {
  const orderKey = 'shared-checkout-order';

  // Anti-pattern: the retry changes the business precondition.
  if (testInfo.retry > 0) await api.deleteOwnedOrder(orderKey);

  await page.goto/orders/new?key=${orderKey});
  await page.getByRole('button', { name: 'Create order' }).click();
  await expect(page.getByTestId('order-key')).toHaveText(orderKey);
});

Expected ledger:

The example’s retry and worker-index transitions are grounded in documented runner semantics: the initial testInfo.retry value is 0, the first retry is 1, and a restarted worker receives a different worker index while its parallel index remains stable.

The expected passing retry would not prove the original requirement under the original state. It would demonstrate that deleting the conflicting record made the second attempt succeed.

Form Competing Explanations

Do not choose a cause merely because the second attempt passed. Write down alternatives and the observation that would falsify each one.

The “insufficient evidence” row is not a cop-out. It is a valid investigation outcome when attempt 0 was not recorded well enough to distinguish a product defect from test-data contamination.

Prove the Fix Without Changing the Assertion

Redesign ownership instead of cleaning only on retry. Generate a run-scoped, test-owned order key, assert the expected initial ownership or absence state before the business action where appropriate, and clean only records belonging to that namespace. Keep the original business assertion unchanged.

A proposed verification design, not a statistical guarantee, is a local policy of ten fresh isolated runs with retries disabled, followed by five cycles at the normal CI worker count for the affected cohort. Add one deliberate interrupted-teardown case to exercise the janitor or lease path. These counts are an engineering sampling choice, not evidence of zero future flakiness.

Fix evidence checklist: same requirement; same assertion; no retry-only cleanup; fresh unique namespace; attempt-0 artifacts present; selected parallel contention exercised; interrupted teardown exercised; and no unexpected records outside the test’s ownership boundary.

Use Retries to Diagnose, Not to Rewrite the Requirement

Retries are useful when they preserve evidence and expose nondeterminism. They become dangerous when the retry path silently changes what the test is proving. Playwright documents access to testInfo.retry and even provides an example that clears server-side state before a retry. That establishes the mechanism; it does not establish that every retry-only state mutation preserves the semantics of your test.

Treat every retry-dependent branch as a review trigger. Ask whether it changes diagnostics or changes preconditions. Collecting an extra log on retry is diagnostic. Deleting an account, changing feature flags, widening eligibility, or bypassing an assertion changes the scenario.

Avoid arbitrary sleeps. A fixed delay can move a race window without establishing the condition the test needs. Prefer an observable readiness contract: response status, DOM state, API state, queue acknowledgment, or another event tied to the requirement.

Timeout increases deserve the same scrutiny. A larger timeout is justified when the product or test environment has an explicit latency requirement that the previous timeout contradicted. “It passed when we changed five seconds to thirty seconds” is not enough; identify the contract that makes thirty seconds acceptable.

Do not turn exception swallowing into resilience. A helper that catches a failed expectation, logs a warning, and continues can transform a meaningful regression into an apparent success. Likewise, removing an assertion that “causes flakes” is not stabilization unless the assertion was demonstrably invalid for the stated requirement.

The useful review question is: what was wrong before the patch? A synchronization defect should be fixed by waiting on the correct observable condition. A resource collision should be fixed by ownership. An environment-contract problem should be fixed or documented at the environment boundary. A real product defect belongs with the product owner. An unexplainable failure remains an evidence problem.

Fix-review checklist (proposed operating model): preserve the first failure next to the patch; identify synchronization versus ownership versus environment cause; state the expected business precondition; keep assertions semantically equivalent; remove retry-only data mutation; prove the fix with retries disabled; and document any justified timeout against an explicit requirement.

Create a Gate That Reports Flaky Tests Honestly

Playwright added failOnFlakyTests in version 1.52. The current API documentation says that, when enabled, Playwright exits with an error if tests are marked flaky; it also documents the --fail-on-flaky-tests command-line option. It is not an automatic default policy, and its use must be qualified against the installed runner version.

For the pinned 1.63.0 scope, a minimal strict CI configuration can look like this:

// playwright.config.ts: @playwright/[email protected]
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 1 : 0,    // Local policy: one diagnostic retry in CI.
  failOnFlakyTests: !!process.env.CI, // Flaky logical tests fail the CI command.
  use: {
    trace: process.env.CI ? 'retain-on-failure' : 'off',
  },
});

This separates two ideas: permit one bounded retry to collect evidence, but do not let an eventual pass automatically satisfy the required release check. Playwright still categorizes a test that fails initially and passes on retry as flaky.

These are organizational policies, not vendor defaults. A high-risk payment assertion and a low-risk cosmetic telemetry check do not deserve an identical exception process merely because both happen to be flaky. Severity, confidence, replacement coverage, and ownership should determine the decision; there is no defensible universal flaky-test percentage in the evidence gathered for this article.

Also separate the Playwright command from CI-platform rerun behavior. A platform-level “rerun failed job” should never erase the original required-check evidence. Store the CI attempt identifier in the ledger and design the release policy so “the latest attempt is green” cannot silently substitute for the approved gate semantics.

Reliable gating is part of broader QA automation engineering skills, but the decisive controls here are concrete: know the installed Playwright version, know the retry budget, know whether flaky outcomes make the command fail, and retain the evidence that explains why.

Gate checklist (proposed local policy): pin the runner; set retries intentionally; verify failOnFlakyTests exists in the installed version; test the CI exit behavior with a synthetic known-flaky fixture; keep required and diagnostic jobs distinct; preserve original job attempts; and require an owner for every exception.

Quarantine with an Owner, Expiry and Replacement Coverage

Quarantine is a controlled exception, not deletion of inconvenient evidence. A quarantined test still represents a requirement. The team is choosing a temporary alternate way to manage that requirement’s release risk while the automation signal is untrustworthy.

A useful quarantine record names the protected requirement, suspected failure class, business impact, first-failure evidence location, compensating check, repair owner, and expiry. It should also say whether the test continues to execute in a non-blocking lane or is skipped entirely.

Do not let a skipped test disappear into the denominator. Playwright’s reporting model includes skipped status; an unexecuted requirement is not equivalent to a pass. If quarantine works by skipping a test, the release record should say which requirement did not receive its normal automated execution.

The repair owner also needs a clear hypothesis, not merely “test is flaky.” A harness-owned test-data collision belongs with the harness or suite owner. A service that violates an agreed readiness contract may belong with the service owner. A genuine product behavior that intermittently violates the requirement belongs with the product engineering owner. When the evidence cannot choose among them, the owner’s first task is to close the evidence gap.

A good re-entry criterion is evidence-based: the root cause is understood; the assertion remains intact; the harness has deterministic ownership; the test passes the agreed fresh-run verification with retries disabled; and selected parallel execution no longer produces unintended cross-test state.

Quarantine checklist (proposed policy): visible label; ticket; owner; expiry; preserved failure artifact; compensating coverage; explicit release risk; no silent assertion weakening; no indefinite skip; and re-entry evidence reviewed by the ownership group that approved the exception.

Measure Attempts and Logical Tests Separately

Retry metrics become misleading when attempts are used as the denominator for logical test quality. Suppose 100 logical tests run with one retry available. Ninety-six pass on the first attempt. Four fail initially; three pass on retry and one fails again. That produces 104 attempts, but still only 100 logical tests.

The useful metrics are defined at both levels:

In this synthetic measurement example, first-attempt pass rate is 96/100 = 96%; flaky-test rate is 3/100 = 3%; exhausted-failure rate is 1/100 = 1%. Counting 103 successful attempts out of 104 and calling that a 99% logical-test “pass rate” would answer a different question because retry attempts have changed the denominator.

Infrastructure setup failures deserve separate treatment where they occur before a logical test meaningfully executes. If a worker cannot start the required execution environment or a disposable prerequisite service never becomes ready, record the event as an environment or setup failure alongside the affected test-execution count rather than silently converting every blocked scenario into a product assertion failure.

Playwright gives you the attempt index through testInfo.retry and defines passed, flaky, and failed logical outcomes in its retry documentation.

Artifact completeness needs its own denominator because a failure without sufficient evidence has different diagnostic value from a failure with a trace, request correlation, and state snapshot. For example, “90% of failed attempts had the required evidence bundle” is actionable. “90% of all tests had traces” is less useful if your actual requirement was only to preserve the failures.

Likewise, measure retry time as a cost rather than converting it into reliability. Retry compute can tell you where instability is consuming CI capacity, but a fast retry does not make the original failure less significant.

Measurement checklist (proposed policy): report logical tests and attempts separately; keep first-attempt success visible; count flaky tests once per logical test; count missing artifacts; report skipped and unexecuted requirements; separate environment setup failure; and track retry compute or time without using retry attempts to improve the apparent logical-test pass rate.

Roll Out the Policy Without Blocking Diagnosis

A reliability gate can fail operationally if it is enabled before evidence quality and ownership are ready. The rollout should make hidden failures visible first, then tighten enforcement as teams remove the causes.

A practical local-policy rollout is 30 days, not because 30 is universally correct, but because it provides an explicit time box for evidence work, fixture repairs, gate trial, and exception cleanup.

During the first week, do not change five variables at once. Capture the existing retry behavior and identify missing evidence. In week two, prioritize failures that protect high-risk requirements, recur enough to obstruct diagnosis, or consume material investigation and retry effort according to your own data. In week three, repair ownership and synchronization while leaving requirement assertions intact. In the final phase, trial the proposed gate before making it a required release control.

The entry artifact for each investigated flake should be small but complete: runner fingerprint, logical test identity, attempt ledger, first-failure artifact references, server build, resource namespace, and current owner. The exit artifact should state the classification: product bug, test defect, environment failure, or insufficient evidence; and the decision: fixed, temporarily quarantined, release held, or returned for evidence collection.

If the new gate is too noisy, roll back enforcement without rolling back visibility. Keep flaky classification, attempt ledgers, and first-failure artifacts active while you refine ownership and exception handling. The failure mode to avoid is restoring “rerun until green and discard the red attempt.”

Engineers who are building an automation portfolio can demonstrate the same reasoning on a smaller project: preserve the original failing attempt, show the ownership or synchronization defect, state competing hypotheses, present the patch, and explain the release-policy decision. That demonstrates diagnostic judgment rather than only framework syntax.

Rollout checklist (proposed policy): establish a baseline before enforcement; assign artifact owners; prioritize by requirement risk; prove fixture fixes with retries disabled; trial the gate; time-box exceptions; preserve visibility during rollback; and review the policy after the first full cycle using concrete failure cases.

Develop the QA Skills Behind Reliable Evidence

Retry diagnosis sits at the intersection of test design, framework behavior, test-data ownership, CI integration, and evidence review. Those skills remain useful even when a specific Playwright option changes across runner versions.

Refonte Learning’s verified QA Automation Engineering Program currently lists a three-month period and 12–14 hours per week. The live program page also names automated test scripts, test automation frameworks, CI/CD pipeline integration, performance and security testing, and practical web/mobile testing work among its subjects and competencies.

Those confirmed foundations are relevant to the engineering discipline in this playbook without implying that a particular Playwright retry option or this exact workflow is part of the curriculum.

Skill checklist: learn to state the protected requirement before changing the test; distinguish browser isolation from backend isolation; identify resource owners; read runner and version documentation; preserve attempt-level artifacts; and explain why a green retry can still justify a fix, quarantine, or release hold.

Resolve the Remaining Retry Questions

A reliable retry policy should leave fewer ambiguous questions after the job finishes. Use this decision table when triaging a green-after-retry result.

Does a retry resume the same worker? No. Playwright documents that after a test failure it discards the worker process and browser and starts a new worker; with retries enabled, the failed test is then retried in the replacement worker.

Is the server reset? Not by the browser-context isolation contract. Playwright documents isolated browser contexts for browser state. Arbitrary external server records and services fall outside that browser isolation boundary and require a harness or environment ownership model.

Does on-first-retry capture the first failure? No. The Trace Viewer guide says it records a trace when retrying a test for the first time. If attempt 0 was not traced, a later retry trace cannot retrospectively recreate unrecorded attempt-0 browser evidence.

Should every flaky test block a release? Playwright provides ,c;39 whether to turn that mechanism into a blocking release policy is an organizational decision. The API says the option exits with an error when tests are marked flaky and documents it as added in v1.52. Risk, replacement coverage, evidence quality, and exception ownership should determine whether you fix immediately, quarantine temporarily, or hold the release.

The final decision should preserve the distinction between four outcomes. Fix when the cause has been reproduced and removed without weakening the protected assertion. Quarantine only as a visible, owned, expiring exception with replacement coverage. Hold when the observed failure plausibly represents a high-risk requirement violation or the release cannot safely absorb the uncertainty under your policy. Insufficient evidence is the correct classification when the original attempt cannot be reconstructed well enough to choose between product, harness, and environment explanations.

The closing rule is simple: execution success is not the same as confidence in the tested requirement. A retry is another attempt, not an eraser. Preserve the first thing the suite told you, identify what changed before the second attempt, and make the release decision from that evidence.