The Exception Test Passed.
Its State Assertion Never Ran.

A negative test can be green for exactly the wrong reason. The code rejects an invalid reservation with the expected exception, pytest reports a pass, and the reviewer assumes the ledger stayed unchanged. Yet the implementation may already have appended a bad row before raising. If the state assertion sits after the raising call but remains indented inside with pytest.raises(...), that assertion is unreachable on the very path the test is meant to validate.

This playbook examines a defect in the test’s pass/fail logic. The bounded fixture is a synthetic in-memory reservation ledger. In a recorded run on October 19, 2026, seven fresh child processes using CPython 3.12.14 and pytest 9.0.2 produced the expected exit statuses, assertion-call counts, and ledger contents. Those observations apply to that fixture and execution; they do not establish results for every Python or pytest release.

The release rule is stricter than “the exception test passed.” ACCEPT only when the expected exception policy and the reachable postcondition both pass, and when a deliberately faulty implementation makes the repaired test fail. REPAIR the assertion scope. REJECT the original false-green result as evidence of state preservation. HOLD when collection, artifacts, traces, or harness behavior cannot establish the intended result. A wrong or missing exception is expected in its designated negative control; elsewhere it contradicts the declared contract. A deliberately failing child can be a successful control for the outer experiment, so child and driver exit statuses must be interpreted separately.

Define the exception-and-state contract before trusting green

The synthetic behavior under test has two independent claims. First, an invalid reservation must be rejected with the declared exception policy. Second, after rejection, the ledger must still be empty. A test that proves only the first claim is not partial proof of the second; it is evidence for a narrower contract.

That distinction matters because an exception is control flow, not rollback. In the faulty branch of this fixture, the implementation appends {"id": "reservation-1", "quantity": -2} and then raises Rejected("quantity must be positive"). The exception can satisfy the configured message match and still arrive after state has changed. The useful review question is which promised observations actually executed before pytest declared success. For broader context, see Refonte Learning’s guide to QA automation roles and skills.

The pytest.raises contract is explicit. Its context-manager form succeeds when the block raises the requested type or a subclass and satisfies any supplied match condition. If the requested exception is not raised, the check fails. The pytest.raises API reference also explains that statements after the raising call within the same context-manager suite are skipped on that exception path. Post-exception assertions belong outside the with.

Python’s with statement semantics explain why. When an exception exits the suite, the context manager’s exit() receives it. If exit() suppresses the exception, execution continues with the statement following the entire with statement. It does not resume at the next indented line that the exception already skipped. The Python 3.12 language reference documents this control flow for the interpreter family used in the recorded run.

The defect can be seen in four lines:

# False green: state check is unreachable after the expected raise.
with pytest.raises(Rejected, match="quantity"):
    reserve(ledger, mode)
    assert ledger == []

The repaired scope is not a different exception policy. It is the same exception assertion followed by a separately reachable state assertion:

with pytest.raises(Rejected, match="quantity"):
    reserve(ledger, mode)

assert ledger == []

Treat the two assertions as two evidence units. The exception observation establishes rejection. The postcondition establishes residual state. Neither substitutes for the other, and neither should be inferred from a green test summary.

A useful evidence taxonomy prevents review language from drifting. “Documented behavior” means the semantics stated by Python or pytest. “Expected result” means the independently authored oracle for a case. “Observed result” means the actual exit code, invocation count, and ledger captured from an execution. “Operational policy” means the team’s ACCEPT, REPAIR, REJECT, or HOLD decision. Mixing those categories produces dangerous sentences such as “pytest guarantees the ledger is unchanged” or “the driver passed, so every child passed.” Neither follows from the evidence.

Build fresh state and run every case in a fresh pytest child

The probe should remove alternative explanations before diagnosing assertion reachability. Every case gets a new pytest process and a new in-test ledger. This prevents shared parameter objects and ordinary in-process module state from carrying over between cases. The driver also controls pytest configuration and plugin loading. These measures isolate the intended experiment without claiming a completely independent operating-system environment. For the separate problem of shared test inputs, see mutable parameter isolation.

The complete driver below writes one temporary test module and a local empty pytest configuration. It selects exactly one test node in each child, prevents conftest.py loading, disables automatic third-party plugin loading, clears inherited pytest additions, and disables the cache provider. It captures stdout and stderr, requests JUnit XML, and records JSON evidence from the test’s finally block. The ledger and reachability counter are allocated inside each test invocation. It also checks exact ledger contents, testcase identity, and the expected failure signature. No real reservation service, payment service, database, or mock is involved.

Prerequisites are a Python environment containing pytest 9.0.2, a writable temporary directory, and permission to launch child processes. Save the driver as raises_probe.py. In an isolated environment, use the interpreter selected for the experiment to install pytest and run the file:

python -m pip install pytest==9.0.2
python raises_probe.py

"""Probe exception/state assertions in seven isolated pytest children."""
import json
import os
from pathlib import Path
import subprocess
import sys
import tempfile
import xml.etree.ElementTree as ET
import pytest
TEST_SOURCE = r'''
import json
import os
from pathlib import Path
import pytest
class Rejected(Exception):
    pass
class DetailedRejection(Rejected):
    pass
def reserve(ledger, mode):
    if mode.endswith("bad"):
        ledger.append({"id": "reservation-1", "quantity": -2})
    if mode == "fixed_wrong_error":
        raise ValueError("quantity must be positive")
    if mode == "fixed_no_error":
        return
    if mode == "fixed_subclass":
        raise DetailedRejection("quantity must be positive")
    raise Rejected("quantity must be positive")
def test_rejected_order_keeps_ledger():
    mode = os.environ["PROBE_CASE"]
    ledger = []
    evidence = {"case": mode, "postcondition_calls": 0}
    def verify_state():
        evidence["postcondition_calls"] += 1
        assert ledger == [], "rejected reservation changed the ledger"
    try:
        if mode.startswith("flawed_"):
            with pytest.raises(Rejected, match="quantity"):
                reserve(ledger, mode)
                verify_state()  # Unreachable after the expected raise.
        else:
            with pytest.raises(Rejected, match="quantity"):
                reserve(ledger, mode)
            verify_state()
    finally:
        evidence["ledger"] = ledger
        Path(os.environ["PROBE_EVIDENCE"]).write_text(
            json.dumps(evidence, indent=2), encoding="utf-8")
'''
# Expected triples: child status, postcondition calls, ledger row count.
EXPECTED = {
    "flawed_bad": (0, 0, 1),
    "flawed_good": (0, 0, 0),
    "fixed_bad": (1, 1, 1),
    "fixed_good": (0, 1, 0),
    "fixed_wrong_error": (1, 0, 0),
    "fixed_no_error": (1, 0, 0),
    "fixed_subclass": (0, 1, 0),
}
FAILURE_PREFIX = {
    "fixed_bad": "AssertionError: rejected reservation changed the ledger",
    "fixed_wrong_error": "ValueError: quantity must be positive",
    "fixed_no_error":
        "Failed: DID NOT RAISE <class 'test_contract.Rejected'>",
}
def main():
    root = Path(tempfile.mkdtemp(prefix="raises-probe-"))
    module = root / "test_contract.py"
    module.write_text(TEST_SOURCE, encoding="utf-8")
    config = root / "pytest.ini"
    config.write_text("[pytest]\n", encoding="utf-8")
    results = []
    for case, expected in EXPECTED.items():
        env = os.environ.copy()
        env.pop("PYTEST_PLUGINS", None)
        env.update(PYTEST_DISABLE_PLUGIN_AUTOLOAD="1", PYTEST_ADDOPTS="",
                   PROBE_CASE=case,
                   PROBE_EVIDENCE=str(root / f"{case}.json"))
        junit = root / f"{case}.xml"
        command = [
            sys.executable, "-m", "pytest", "-q",
            "-c", str(config), "--noconftest",
            "-p", "no:cacheprovider",
            "test_contract.py::test_rejected_order_keeps_ledger",
            f"--junitxml={junit}",
        ]
        run = subprocess.run(command, cwd=root, env=env, text=True,
                             capture_output=True, timeout=30, check=False)
        (root / f"{case}.stdout.txt").write_text(run.stdout, encoding="utf-8")
        (root / f"{case}.stderr.txt").write_text(run.stderr, encoding="utf-8")
        evidence = json.loads(
            Path(env["PROBE_EVIDENCE"]).read_text(encoding="utf-8"))
        cases = ET.parse(junit).findall(".//testcase")
        if len(cases) != 1:
            raise RuntimeError(f"HOLD: testcase count in {case}")
        node = cases[0]
        if (node.get("classname") != "test_contract"
                or node.get("name") != "test_rejected_order_keeps_ledger"
                or node.find("skipped") is not None
                or node.find("error") is not None):
            raise RuntimeError(f"HOLD: testcase identity/status in {case}")
        failures = node.findall("failure")
        prefix = FAILURE_PREFIX.get(case)
        if prefix is None:
            valid_failure = not failures
        else:
            valid_failure = (len(failures) == 1 and
                failures[0].get("message", "").startswith(prefix))
        if not valid_failure or not run.stdout.strip():
            raise RuntimeError(f"HOLD: failure signature/trace in {case}")
        expected_ledger = (
            [{"id": "reservation-1", "quantity": -2}]
            if expected[2] else []
        )
        if (evidence["case"] != case or
                evidence["ledger"] != expected_ledger):
            raise RuntimeError(f"HOLD: case/state evidence in {case}")
        observed = (
            run.returncode,
            evidence["postcondition_calls"],
            len(evidence["ledger"]),
        )
        if observed != expected:
            raise RuntimeError(f"HOLD: {case}: {observed} != {expected}")
        results.append({**evidence, "exit_code": run.returncode,
                        "command": command})
    report = {
        "python": sys.version,
        "pytest": pytest.__version__,
        "evidence_directory": str(root),
        "environment_controls": {
            "PYTEST_DISABLE_PLUGIN_AUTOLOAD": "1",
            "PYTEST_ADDOPTS": "", "PYTEST_PLUGINS": "unset",
        },
        "cases": results,
        "probe_status": "EXPECTED_OUTCOMES_MATCH",
    }
    (root / "report.json").write_text(
        json.dumps(report, indent=2), encoding="utf-8"
    )
    print(json.dumps(report, indent=2))
if name == "__main__":
    main()

Several controls in that driver deserve separate attention. sys.executable with -m pytest launches each child using the driver’s interpreter. The exact node ID limits selection, and the JUnit identity check verifies which test was reported. The local pytest.ini, --noconftest, and documented plugin-loading controls reduce inherited pytest configuration effects: automatic third-party discovery is disabled, PYTEST_PLUGINS is removed, and PYTEST_ADDOPTS is cleared. Plugin autoload suppression alone would not disable plugins explicitly named through PYTEST_PLUGINS. Fresh processes prevent ordinary import caching and in-memory state from an earlier child from carrying into a later one.

JUnit XML and JSON answer different questions. The driver requires exactly one JUnit testcase with the expected identity, no skip or error result, and a failure element only for the designated failing controls. The JSON records the case name, exact synthetic ledger, and number of postcondition calls. Stdout and stderr preserve the child’s diagnostics. These artifacts must agree: a passing process status can still be the false green being investigated, and a matching row count alone cannot establish that the expected row was captured.

The finally block is deliberately diagnostic. It records state even when pytest is failing the test. That makes the pre-cleanup observation visible without turning the evidence write itself into a state assertion. It is also why the counter increments before assert ledger == []: in fixed_bad, the assertion fails, but the counter still proves the postcondition was reached once.

The seven-case execution on CPython 3.12.14 with pytest 9.0.2 matched every declared result triple and exact ledger expectation. The three deliberately failing children also produced their expected JUnit failure-message prefixes. These are observations from that execution, not claims that those versions are the newest releases or that diagnostic wording is stable across all versions. Review a changed diagnostic before updating FAILURE_PREFIX.

A useful artifact review starts from the temporary evidence directory. For each case, pair CASE.json with CASE.xml, CASE.stdout.txt, and CASE.stderr.txt, then reconcile them with the entry in report.json. The JSON records state and reachability; JUnit records testcase identity, status, and failure information; stdout and stderr preserve diagnostics; the report records the commands and successful comparisons. An empty stderr file is normal when the child has nothing to write there. A missing artifact or an unexplained disagreement is HOLD because the evidence cannot establish the intended control path.

Treat each later execution as a separate observation. Record its interpreter, pytest version, command line, environment controls, and evidence directory. A matching rerun increases confidence under the new conditions; it does not change what a previous run established. Conversely, a mismatch is not permission to edit EXPECTED or a diagnostic prefix merely to recover green. Determine whether the implementation, framework, environment, or harness changed.

Prove the false green, then move the postcondition across the boundary

Start with the case that most directly advertises the defect: flawed_bad. The implementation mutates the ledger because the mode ends in "bad", then raises Rejected("quantity must be positive"). The pytest.raises(Rejected, match="quantity") condition is satisfied. Because the exception exits the with suite, verify_state() is skipped. The child therefore returns 0 even though the ledger contains one invalid reservation.

The expected evidence triple is (0, 0, 1): child exit 0, zero postcondition calls, one ledger row. That triple is stronger than “the ledger was dirty after a passing test.” It proves both the false-green outcome and the specific reachability defect. A reviewer does not have to infer that the assertion was skipped; the counter immediately before the assertion stayed at zero.

Now change only the test scope, not the faulty implementation. In fixed_bad, reserve() still appends the invalid row and still raises the expected Rejected. The with block ends immediately after the raising call. Once pytest.raises suppresses that expected exception, control resumes after the with, where verify_state() is reachable. The counter moves to one and assert ledger == [] fails. The child returns 1, and the final JSON still records one ledger row.

That expected (1, 1, 1) outcome is the decisive mutation-style control. It shows the repaired oracle can kill the known defect. A state assertion that is merely present in source code is not enough; it must be reachable on the rejection path and sensitive to the advertised state mutation.

Only after that control should fixed_good be considered. Here the implementation does not append because the mode does not end in "bad"; it raises the expected Rejected, the context manager completes normally, and verify_state() runs once against an empty ledger. The expected (0, 1, 0) triple demonstrates both pieces of the contract under the good implementation.

This ordering matters operationally. If a team runs only fixed_good and sees green, the repaired indentation looks plausible but has not yet shown that it can detect the targeted defect. The deliberately faulty fixed_bad case is a test of the test. It belongs in validation evidence, not in production code and not as an xfail-masked release test.

Keep flawed_good as a comparison. Its (0, 0, 0) result shows why a clean final ledger does not rehabilitate the original test. The implementation happened to satisfy the postcondition, but the promised state assertion still never ran. That case separates “property happened to be true” from “test actually verified property.” The false-green defect exists in both flawed cases; flawed_bad merely makes its risk visible.

The acceptance rule can therefore be written precisely: ACCEPT the repaired state contract only when fixed_good proves the expected exception plus a reachable clean-state assertion, and fixed_bad proves the same assertion can fail against the known mutation. REPAIR when the assertion remains inside the raising suite. REJECT the earlier flawed_bad pass as evidence of state preservation. A green exception check can remain evidence for the exception claim, but it cannot be promoted to evidence for the ledger claim it never executed.

Exercise wrong-error, no-error, and subclass controls
without weakening the oracle

The repaired scope needs exception-policy controls as well as a state-mutation control. Otherwise a postcondition could be sound while the raises clause is too permissive, too narrow, or simply not exercised as intended.

fixed_wrong_error raises ValueError("quantity must be positive"). The message still contains "quantity", but the type is outside the declared Rejected hierarchy. pytest.raises(Rejected, match="quantity") therefore does not satisfy the contract. The unexpected ValueError propagates out of the context, verify_state() is never reached, the evidence file is still written by finally, and the child is expected to return 1 with (1, 0, 0). This is why a recognizable message is not enough.

fixed_no_error returns from reserve() without raising. When the with suite ends normally, pytest.raises itself fails because the expected exception never occurred. verify_state() after the context is therefore not reached, and the expected evidence is (1, 0, 0). The pytest assertion documentation explains expected-exception checks and how they differ from state assertions performed after the context.

fixed_subclass intentionally raises DetailedRejection, which subclasses Rejected. Under pytest’s documented subclass policy, pytest.raises(Rejected, ...) accepts that subclass unless the test adds an exact-type assertion. The expected triple is (0, 1, 0): the subclass satisfies the declared rejection type, the postcondition runs, and the ledger remains empty. This is an explicit exception-policy choice.

A team that requires exact exception identity should state that additional contract separately. Capture the context result with with pytest.raises(Rejected) as exc_info:, call the operation inside the suite, and then assert exc_info.type is Rejected after the context. Do not interpret subclass acceptance as exact-type proof. Add the narrower requirement only when it reflects the behavior being promised.

The same discipline applies to the message. In this fixture, match="quantity" uses regular-expression search semantics against the formatted exception text. Pytest can include attached exception notes in that text; this fixture adds no notes. The match establishes that the declared fragment is present, not that the complete message is identical.

Avoid “fixing” a wrong-error control by broadening the expected type to Exception. The pytest API warning about broad exceptions explains that this can hide real bugs because unrelated exception types may satisfy the context manager. Here the rejection hierarchy is meaningful: Rejected and its subclass are acceptable; ValueError is not.

These controls produce an important separation of failure paths. If the wrong type propagates, the postcondition counter remains zero because the context manager did not successfully handle the declared rejection. If no exception occurs, the counter also remains zero because pytest.raises fails at context exit before execution can proceed to the next statement. A zero counter is therefore not automatically the original defect. It must be interpreted together with child exit status, stderr/stdout, JUnit, and the case’s expected exception behavior.

The decision logic should never reduce to “counter must equal one in every case.” It is “counter must equal the independently expected reachability for this control.” In the accepted business path fixed_good, the postcondition must run once. In the deliberately faulty fixed_bad, it must also run once and fail. In wrong-error and no-error controls, it should remain zero because the exception contract itself failed first. That is why the expected table is written before the children are launched.

Separate child exit codes, harness health,
evidence capture, and cleanup concerns

The pytest 9.0 exit-code reference defines the statuses relevant to the tested pytest 9.0.2 runtime: 0 means the collected tests passed; 1 means tests ran and at least one failed; 2 indicates interruption; 3 indicates an internal error; 4 indicates a command-line usage error; and 5 means no tests were collected. The rolling stable exit-code reference additionally documents code 6 for exceeding the configured maximum warnings. That extra code is absent from pytest 9.0.2’s ExitCode enumeration and is not exercised by this probe.

For this probe, child exit 1 is not automatically a bad experiment result. fixed_bad, fixed_wrong_error, and fixed_no_error are deliberately expected to fail as child tests. The driver treats them as correct controls only when their result triples, exact state, testcase identity, and failure signatures match the declared expectations. After all seven cases satisfy those checks, it prints "probe_status": "EXPECTED_OUTCOMES_MATCH". The outer experiment can therefore exit successfully because some children failed in the intended way.

That distinction is easy to lose in CI. A dashboard that only shows the outer job as green can conceal whether the negative controls actually failed, while a dashboard that treats any captured child exit 1 as a harness failure can reject a correct experiment. Preserve the case-level report, not just the parent process status.

The probe guards against a misleading interpretation of exit 0 by checking JUnit separately. It requires one <testcase> element with the selected test identity and no <skipped> or <error> child. No collected tests is a distinct exit condition, and a skip does not exercise the advertised state oracle. Unexpected collection, interruption, invocation problems, or internal errors are HOLD conditions rather than evidence that the behavioral contract passed.

HOLD also covers missing evidence. The driver explicitly raises RuntimeError("HOLD: ...") for its testcase, failure-signature, trace, case-identity, state, and result-comparison checks. Other harness faults, such as subprocess.TimeoutExpired, missing or malformed JSON, absent or unparsable JUnit XML, or an unwritable artifact directory, can surface as other Python exceptions. They still require HOLD because the evidence is incomplete. A timeout may prevent the normal stdout and stderr files from being written; it must not be converted into a behavioral pass.

Preserve stdout and stderr for every completed child, including intentionally failing controls. The driver checks the expected JUnit failure-message prefix so that a different exception cannot silently satisfy an identical numeric result triple. Read the full trace as well: a prefix is a useful guard for this fixture, not proof of every possible failure mechanism. The expected prefixes are tied to the tested pytest version, so a future formatting change requires review.

This is also where state validation must be kept separate from cleanup. The test’s finally block writes evidence; it does not restore the ledger and does not assert that cleanup happened. In a real resource-owning fixture, teardown policy would require its own design. That is intentionally outside this lab; see cleanup ownership when pytest setup fails for that neighboring concern.

A diagnostic finally runs during ordinary exception unwinding, including when an assertion fails; Python’s try statement semantics describe the underlying rule. It is not a guarantee that evidence will be written after forced process termination or a failed write. The observation point also matters. If cleanup emptied a shared store before evidence was captured, a post-cleanup “empty” observation could hide the earlier mutation. The property checked here is the reachable postcondition immediately after the expected rejection and before any hypothetical cleanup. Recording diagnostics and proving cleanup correctness are separate obligations.

Review assertion reachability and apply the acceptance matrix

A code review can detect this defect without a large testing philosophy. Follow the control-flow boundary of the negative path. Locate the call expected to raise. Mark the exact with pytest.raises(...) suite. Then list every claim the test name, comments, ticket, or reviewer expects the test to establish after rejection. Any postcondition placed textually after the raising call but still inside that suite is suspect because the expected exception exits the suite before that line.

A compact review procedure is enough:

1.   Identify the single operation expected to raise and keep the raises scope as narrow as practical.

2.   Write the exception contract explicitly: accepted type hierarchy and any message rule.

3.   Place residual-state assertions after the with, not after the raising call inside it.

4.   Make reachability observable when validating a suspected false green; a counter immediately before the postcondition is sufficient for this synthetic probe.

5.   Inject one controlled implementation fault that violates the state invariant while preserving the expected exception. The repaired test must fail for that fault.

6.   Preserve the exact child node, status, stdout, stderr, JUnit result, and state evidence. HOLD if the artifacts do not establish what ran.

That procedure is a focused mutation probe, not a request to adopt a general mutation-testing platform. The key is causal sensitivity: can the advertised bad state make the repaired oracle fail while the exception behavior remains unchanged?

The release decision matrix should be explicit:

Do not “repair” a failing fixed_bad control by changing its expected result to green. That would erase the test-of-the-test. A changed expected outcome is itself a review event: either the synthetic implementation changed, the contract changed, or the harness no longer measures the same thing. Restoring CI color is not the objective; preserving diagnostic meaning is.

Keep xfail/XPASS governance separate from this experiment. The failing child is an intentional control launched by the driver, not a known production defect being waived in the release suite. For the separate marker policy, consult pytest’s expected-failure and unexpected-pass documentation.

The matrix also prevents overclaiming. ACCEPT means “this repaired synthetic contract, under the recorded invocation, demonstrated both exception handling and state preservation, and its known-fault control was detected.” It does not mean the application has transaction rollback guarantees, that every rejection path is side-effect free, that every plugin combination is safe, or that all future pytest/Python versions behave identically. Those broader claims require broader evidence.

A practitioner review should also distinguish source presence from execution. Seeing assert ledger == [] in a diff is not evidence. Seeing it fail in fixed_bad is evidence of reachability and sensitivity. Seeing it pass in fixed_good is evidence that the good implementation satisfies the synthetic postcondition. Seeing it never run in flawed_good proves why a clean state by coincidence is still insufficient test evidence.

Requalify affected failure paths and assign evidence ownership

Once one unreachable postcondition is found, search adjacent negative tests for the same structural pattern: with pytest.raises(...), a call intended to raise, then additional assertions at the same indentation before the context closes. Focus on tests that promise a post-exception property, such as unchanged collections or counters, no emitted record, or a preserved cache entry, but place that check after the expected raise inside the same suite.

Requalification should happen from known fresh inputs. For each affected path, separate evidence repair from application repair. First move the assertion to a reachable location without changing the implementation. If the test now fails, that failure may reveal a pre-existing side effect that the old false-green test had been hiding. Do not immediately weaken the assertion or rewrite the expected state to recover green. Preserve the failing output and state snapshot, then decide whether the application behavior or the contract is wrong.

The same fault-injection principle should follow the repair. Construct the smallest controlled mutation that preserves the expected exception while violating the residual state invariant. The repaired test should reject it. This is especially valuable when the original false-green test has been passing for a long time: historical greenness does not establish that the state clause was ever exercised.

Ownership should be divided by claim. The backend maintainer owns the behavioral contract: what “rejected” means and which residual state is permitted. The test engineer owns oracle reachability and detection evidence: whether every claimed postcondition executes on the intended path and can fail under a controlled violation. The reviewer or release owner owns disposition: ACCEPT, REPAIR, REJECT, or HOLD based on the preserved artifacts rather than on a summary count.

Retain case-level evidence in CI. The useful unit is the selected node plus its environment and artifacts: interpreter version, pytest version, command, case name, child return code, postcondition call count, ledger contents, JUnit testcase state, stdout, and stderr. The outer report.json is a convenient index, but it should not replace the underlying traces. A release record that says only “7 cases checked” loses the distinction between a correct child failure and a harness failure.

When placing these checks into delivery workflows, make evidence checks explicit validation steps. Refonte Learning’s overview of DevOps pipeline practices provides broader delivery context. The operational requirement here is to preserve enough detail for each case to show that the expected node ran under the declared command and produced the expected state, reachability, and failure evidence.

Version labels deserve the same discipline. The recorded runtime is CPython 3.12.14 with pytest 9.0.2. The Python 3.12 and pytest 9.0 documentation families support the relevant semantics, while rolling documentation may describe additional features. Keep runtime observations separate from later documentation updates. Neither a new page timestamp nor a matching code snippet establishes that a particular interpreter was executed.

Keep the synthetic data boundary intact. The reservation row is test input created in memory and preserved as diagnostic evidence. Nothing here demonstrates recovery of unknown remote side effects, rollback of production transactions, or cleanup of external resources. The experiment isolates the test’s ability to verify its postcondition: did the assertion run on the expected rejection path, and could it detect the deliberate mutation?

Develop stronger negative-test evidence with Refonte Learning

The durable habit is simple: test rejection and residual state as separate claims. Put only the operation expected to raise inside pytest.raises; put post-exception state assertions after the context; then prove the repaired oracle can fail when a controlled implementation fault changes state before raising. Preserve the failing control as validation evidence, not as an xfail-masked production test.

For the recorded fixture, flawed_bad is a false green and must be REJECTED as state evidence. The assertion scope must be REPAIRED. fixed_bad must fail with one postcondition call and the expected dirty ledger row, while fixed_good must pass with one postcondition call and an empty ledger. The wrong-error and no-error controls must fail in their separately declared ways. Unexpected skips, missing collection, missing artifacts or traces, timeouts, and contradictory results require HOLD.

The recorded CPython 3.12.14 / pytest 9.0.2 execution demonstrates this control-flow defect and the repaired test’s sensitivity within the synthetic in-memory ledger. It does not establish general rollback behavior or production safety.

For structured study in test engineering and automation, review Refonte Learning’s Quality Assurance Automation Engineering / Quality Assurance Engineering Program. Its published page describes a three-month format at 12–14 hours per week, automated testing and test frameworks, CI/CD integration, performance and security testing, project work, mentor guidance, and potential internship opportunities. It also lists a Training Certificate and a Certificate of Internship upon successful completion. Confirm current eligibility and internship arrangements with Refonte Learning before enrolling.