PyTorch DataLoader Reused Its Workers. Did the Next Epoch Get New Randomness?

Modern PyTorch allows a DataLoader to reuse (“persist”) its worker processes across epochs. This raises a subtle question: when persistent_workers=True, does each new epoch still get fresh randomness? We analyze this for two separate randomness domains: the per-epoch sampler shuffle and the per-worker augmentation RNG. We proceed like a software release validation “playbook.”

First we define a precise contract: what behavior we expect and how to test it. We then pin an isolated fixture using a synthetic dataset and exact PyTorch versions (e.g. PyTorch 2.15.0). We capture the documented baseline from PyTorch docs: persistent_workers keeps worker processes alive, and the official reproducibility notes say DataLoader reseeds workers via worker_init_fn and generator. We build a simple fixture and an expected-case manifest that lists scenarios (with and without persistence, shuffle, worker_init_fn) and their expected behavior. For each test case, we maintain a per-case evidence ledger tracking commands and outputs.

We then reproduce the misleading “success” case where a naive check passes but state leaks, and run a comparison case side-by-side (persistent vs non-persistent) to reveal differences. Throughout we exercise failure boundaries (e.g. multiple epochs, seeding variations) and reconcile state vs spec. We keep documentation separate from observed data and identify any discrepancy (“repair boundaries”). We propose minimal repairs if needed, verify that fixes can be rolled back safely, and finally decide ACCEPT / REPAIR / RERUN / HOLD based on evidence.

This disciplined approach mirrors methods in machine learning model evaluation (e.g. ML model evaluation contracts) and other playbooks (async context capture, Kubernetes preStop, HTTP If-Match, CSS transitions), but here we focus exclusively on PyTorch DataLoader RNG behavior. We cite PyTorch docs and source for all claims, use exact version pins, and mark any unrun test as EXPECTED/NOT EXECUTED. In the end we make a clear release decision.

1. Define the Acceptance Contract

We ask: When a PyTorch DataLoader with persistent_workers=True finishes one epoch and starts the next, do both the sample shuffle and any worker-side randomness start afresh? In other words, are each epoch’s shuffles and augmentations independent? Our contract splits this into two assertions:

• Per-epoch shuffle: If shuffle=True (and a given RNG generator is set), each epoch’s index order should differ (given fixed seed we still expect new permutations). The DataLoade docs say the given generator is used by the RandomSampler to generate indices and to draw a base seed for workers. In practice, we expect the order of samples to change each epoch unless we fix seeds or comparisons show otherwise.

• Worker-side RNG: If the Dataset or worker_init_fn uses randomness (e.g. calls torch.rand* or random.seed() in getitem), then starting a new epoch should reset or reseed each worker’s RNG. The official reproducibility notes say DataLoader will reseed workers according to its algorithm. With persistent_workers=True, we must verify that this reseeding still occurs each epoch and isn’t skipped.

These two dimensions yield cases: (persistent or not) × (shuffle or not) × (use worker_init_fn or default RNG). Each test will run multiple epochs (e.g. 3) and record outputs. The acceptance criteria are:

1. Sampler randomness: Index orders in each epoch match the pattern expected when using the given generator. Specifically, if we manually seed the DataLoader’s generator, two consecutive epochs should show different orderings (if shuffle is True) but reproducibly so between runs with identical seeds.

2. Worker RNG: If no manual seeding is done by us, then persistent workers would continue from old state, which could be a bug. We require that each epoch’s worker RNG state is deterministically reproducible (with proper seeding). In other words, under our fixture with a worker_init_fn that uses torch.initial_seed(), worker outputs should match the “expected” independent sequence.

We formally define our contract in an “expected-case manifest” (next section) and a decision rule: only if all evidence (for shuffle and worker RNG) matches the documented intent do we ACCEPT. Any deviation triggers a REPAIR or HOLD. This mirrors the practice of writing a contract before evaluation.

For example, if a reference DataLoader run (non-persistent, with generator) yields sample orders [A,B,C] in epoch 0 and [B,C,A] in epoch 1, then with persistence on we should see a similar independent shuffle pattern. If instead we see [A,B,C] again (duplicated), that would violate our contract. Similarly, if a worker doing return random.random() in each getitem produces 0.7 and 0.4 in epoch 0, we expect new values (like 0.2 and 0.5) in epoch 1 (given a fixed seeding strategy). All expected vs observed behaviors are laid out below in a manifest.

Related work: The Refonte playbook on machine learning evaluation stresses writing a clear testable contract before checking results. Here we likewise specify exactly what evidence counts as success for DataLoader randomness. Unlike ML model metrics, our focus is operational correctness of a library feature.

2. Capture a Pinned Baseline

To avoid any uncertainty, we pin exact versions of all software. For example, we use PyTorch 2.15.0 (the latest stable at writing), Python 3.10, Linux (Ubuntu 22.04) as the environment. This is declared in our fixture. For reproducibility, we show the commands (marked EXPECTED). For instance, to pin PyTorch:

# Pin PyTorch and related libraries in a clean virtual environment
pip install torch==2.15.0 torchvision==0.18.0  # or the exact matched versions
pip install numpy==1.26.2

We include a check snippet:

python - <<EOF
import torch, numpy as np
print("PyTorch version:", torch.__version__)  # EXPECTED: 2.15.0
print("NumPy version:  ", np.__version__)    # EXPECTED: 1.26.2
EOF

Running this (in our analysis) would verify the pinned versions. Pass/Fail: It fails if torch.__version__ is not “2.15.0” or if any import fails. (In our write-up we label outputs EXPECTED because we assume the intended environment is met.) This ensures test reproducibility: if a future run yields different versions, we should RERUN or HOLD.

Pin Fixture Versions

The fixture’s pass condition is “actual versions match specified pins.” For example, if a script prints a PyTorch version other than 2.15.0, that indicates the fixture is not set up as documented (FAIL).

The documentation baseline is taken as current PyTorch docs and source code. For example, the DataLoader API specifies persistent_workers=False by default and describes its effect. We record that as the documented baseline: persistent workers should keep worker processes alive between epochs. We will test if observed behavior aligns with this documentation. If we spot contradictions, we’ll note them explicitly.

3. Create an Isolated Fixture

We build a minimal synthetic dataset to isolate randomness. For example, consider:

import torch
from torch.utils.data import Dataset, DataLoader

# A simple map-style dataset with deterministic content.
class SeqDataset(Dataset):
    def init(self, length):
        self.data = list(range(length))
    def len(self):
        return len(self.data)
    def getitem(self, idx):
        # Simulate a worker-side random augmentation
        val = self.data[idx]
        noise = torch.rand(1).item()  # uses worker RNG
        return val, noise

dataset = SeqDataset(10)

This dataset returns (index, noise) so we can see both index order and randomness. We will run a DataLoader over it. We set num_workers=2 (for real multiprocessing) and choose shuffle=True or False per case. A worker_init_fn may be provided to seed each worker (e.g. using worker_id). This fixture is run in a disposable environment: it has no side-effects, uses only local Python libs, no external data. It is analogous to the async context fixture in Refonte’s playbook: we intentionally minimize extraneous variables and use synthetic inputs so results are attributable solely to DataLoader’s behavior.

We explicitly list the steps to run tests:

• Launch the fixture (say python test_dataloader.py) with the pinned PyTorch. This script iterates over the DataLoader for several epochs, printing out the sequence of (index, noise) pairs for each batch and epoch.

• The environment is reset between major tests to avoid state carryover (a clean process invocation for each case).

• All randomness control (e.g. torch.manual_seed, np.random.seed) is done at the fixture start to produce deterministic behavior when expected.

For example (in our documentation, not executed here), the command would look like:

python test_dataloader.py  # EXPECTED: prints epoch-wise output as per manifest

We ensure no other Python modules interfere. This approach matches the “one-process, deterministic cases” style of other playbooks.

4. Write the Expected-Case Manifest

We enumerate test scenarios and expected outcomes in a table. Each row is a case with a DataLoader configuration and the observed behavior we expect if the contract holds. For example:

Each case’s “Expected Behavior” is determined before execution. This is our manifest. For example, in Case C (persistent, no shuffle), the documented behavior does not explicitly reset workers, so we expect noise to continue (or at least be reproducible in sequence) rather than restart. In Case E, we explicitly insert a reseed, so the noise should restart each epoch.

Negative Control

We include a trivial “no randomness” scenario: e.g. persistent_workers=True, shuffle=False, and a dummy dataset without any randomness. In this control, both epochs must produce identical output (index and values). For example, using shuffle=False and a constant dataset, we assert that outputs from epoch 0 and epoch 1 are exactly the same. Pass/Fail: If they differ, our test fixture or environment is flawed (FAIL).

The manifest table is independent of any actual run: it represents what correct behavior looks like. We treat any differing observation as evidence. At runtime, we will compare actual logs against these expectations.

5. Reproduce the Misleading Success

Before comparing cases, we demonstrate the false-positive signal. For instance, one might run a quick test and see no errors:

# Pseudocode to simulate misleading success
dl = DataLoader(
    dataset, batch_size=5, shuffle=True, num_workers=2,
    persistent_workers=True
)
for epoch in range(2):
    for x, noise in dl:
        print("Epoch", epoch, "Data:", x.tolist(), "Noise:", noise.tolist())

A naive observer could see two epochs of output and conclude “both ran” (SUCCESS). But without careful analysis, they might miss that the second epoch reused RNG state. We explicitly reproduce this by running the fixture with persistent_workers=True and no special seeding. The expected misleading output (unverified by us here) would show a valid-looking second epoch. We label it EXPECTED / NOT EXECUTED in documentation. The crucial point: just because DataLoader ran without exception does not mean it met the randomness contract.

We note that such outputs alone are insufficient evidence. The “success” is superficial: the program produces batches as intended, but what about internal state? This mirrors other cases (e.g., CSS transitionend vs cancel, where a finished transition might look okay but actually was cancelled). Our playbook will dig into that hidden state.

6. Run the Comparison Case

Next, we conduct a controlled comparison. We run two parallel experiments:

1. Baseline (non-persistent) DataLoader: No persistent_workers, with a fixed generator seed.

2. Test (persistent) DataLoader: Same parameters but with persistent_workers=True.

For each, we collect outputs over multiple epochs. For example:

# Comparison test: Persistent vs Non-persistent
python - <<EOF
from torch.utils.data import DataLoader, TensorDataset
import torch

# Setup dataset and generator
data = torch.arange(10)
dataset = TensorDataset(data)
g = torch.Generator().manual_seed(42)  # fixed seed

# Baseline DataLoader (no persistent workers)
dl1 = DataLoader(
    dataset, batch_size=5, shuffle=True, num_workers=2, generator=g
)
print("Baseline Run:")
for epoch in range(2):
    print(f"Epoch {epoch}")
    for batch in dl1:
        print(batch[0].tolist())  # indexes
# (Simulated output is EXPECTED / NOT EXECUTED)

# Test DataLoader (persistent workers)
dl2 = DataLoader(
    dataset, batch_size=5, shuffle=True, num_workers=2,
    persistent_workers=True, generator=g
)
print("\nTest Run (Persistent):")
for epoch in range(2):
    print(f"Epoch {epoch}")
    for batch in dl2:
        print(batch[0].tolist())
EOF

Expected: The printed index sequences for each epoch in the two runs. According to our manifest, both should show two different permutations (since generator=42 was used for both). We expect Baseline and Persistent sequences to match if the generator state evolves identically. We annotate the expected outputs as EXPECTED in our records. If they differ, something is wrong (FAIL).

After the run, we check invariants. For example:

• Invariant: Both DataLoaders must cover the same index set {0..9} each epoch, just in different order.

• Invariant: The sequence from epoch0 to epoch1 should differ (because shuffle=True). If persistent worker reuse somehow “stopped” the sampler RNG, the epoch1 in dl2 might erroneously match epoch0.

By comparing side-by-side, we catch subtle discrepancies. This step is like the Kubernetes preStop playbook comparing Pod states. Here, we “assert comparison invariants” (e.g. “no duplicates, no missing indices, different sequences”) to validate the runs.

Assert Comparison Invariants

Each DataLoader epoch should return all 10 elements (index 0–9). If any index repeats or is missing, it fails. Also, since we seeded the RNG, epoch0 vs epoch1 order should be different (PASS) rather than identical (FAIL).

7. Exercise Failure Boundaries

We now push the fixture to reveal any hidden failures:

• Multiple epochs: Run 5+ epochs to see if the pattern breaks over time. Does the sequence eventually repeat unexpectedly?

• Varying seeds: Try different Generator seeds, or even no generator (allowing default seeding). If behavior changes unpredictably, that’s a red flag.

• No shuffle, only augmentation: Test shuffle=False to isolate worker-side RNG. For example, remove shuffle and examine the “noise” values that come from each worker.

• Using worker_init_fn: If persistent workers reuse state, a common fix is to provide worker_init_fn that reseeds. We test with and without it to see the difference.

For instance, test code snippet:

# Negative test: Workers with default RNG, persistent=True, no shuffle
python - <<EOF
import torch
from torch.utils.data import DataLoader, Dataset

class RandomDataset(Dataset):
    def getitem(self, idx):
        # Returns one random number each call
        return torch.rand(1).item()
    def len(self):
        return 4

# Persistent without shuffle and without worker_init_fn
dl = DataLoader(
    RandomDataset(), batch_size=2, num_workers=2,
    persistent_workers=True
)
for epoch in range(3):
    print("Epoch", epoch, [x for x in dl])
EOF

We expect that if no reseeding is done, epoch 1 may continue from epoch 0’s RNG state. The manifest for this “failure-boundary” check: if two epochs produce correlated (not fresh) values, that confirms the suspected leak. In contrast, if we add worker_init_fn=seed_worker (per docs), epochs should reset.

This probes the “edges” of our contract. If any failure appears (e.g. duplication of sequences, or differences from expectations), we record it as evidence of a bug. All outputs and errors are logged for the evidence ledger.

8. Capture Per-Case Evidence

We compile a ledger table of each test case executed, with evidence from the run. The illustrative entries below are EXPECTED / NOT EXECUTED:

Evidence notes: For each, we capture relevant lines (e.g. “Epoch 0: …”, “Epoch 1: …”). Any deviation from expected is a FAIL entry. We preserve these raw logs (in artifacts/evidence.json, as in CSS transitions playbook). In a real run, this would be collected as test output; here we simulate and label them EXPECTED.

Preserve Failure Artifacts

All evidence (scripts, logs, random seeds) must be saved in a way that cannot be altered by normal operations. For example, we would save the stdout of each test to a file. The pass condition is that the evidence for any failing case is non-empty and findable. If a log is missing or incomplete, that test fails our verification.

9. Reconcile State and Identity

We now interpret results versus the documented state. For each case, we note:

• State observed: e.g. “Epoch 0: indices [2,0,1,…], Epoch 1: [9,5,…]” and noise sequences.

• Identities: Tag each output with the epoch and worker ID (internally, PyTorch uses torch.initial_seed() for each worker). We can extract worker_info.id inside worker_init_fn to tie noise to a specific worker instance.

We then align these with the expected manifest. We check invariants like “each data index appears exactly once per epoch” and “each epoch has same length”. We explicitly state any mismatch: e.g. “In Case C, the second-epoch noise values [0.62, 0.19] are not fresh but continue from epoch 0’s sequence [0.42, 0.18]” (hypothetically). If documentation promised reseeding but we didn’t see it (or vice versa), we highlight it.

This is similar to HTTP If-Match or other playbooks’ state reconciliation. For example, in the HTTP If-Match guide they ensure the final document version matches the expected ETag. Here, our “state” is the RNG state. We compare actual RNG state transitions against what PyTorch documentation implies (through generator drawing base seeds). If everything matches (i.e. each RNG base_seed was drawn from the shared generator as per code and worker torch.initial_seed() was refreshed), then the implementation aligns with its spec. Otherwise, we have identified a deviation.

For example, suppose we observe that in Case C (persistent, no shuffle), epoch1’s first index came from the same base_seed as epoch0. That would indicate no new base_seed was drawn at the start of the second epoch. We would reconcile this against the DataLoader iterator source, which shows self._base_seed = random_(generator) only in the iterator’s init (first epoch) but not in reset() (no new baseseed on reset). If the code truly lacked re-seeding on reset, that suggests either the docs are incomplete or an implementation bug.

10. Separate Documentation from Observation

We carefully document what the official sources say and what we observed, without merging them. For example, PyTorch’s DataLoader doc explicitly says persistent workers “remain alive” but does not clearly promise fresh randomness each epoch. The reproducibility note implies a seeded algorithm, but may not mention persistent explicitly. If we find that persistent workers do not reseed by default, that would be a gap.

We place these side-by-side. For instance:

• Doc (DataLoader API): “generator is used by RandomSampler and to generate base_seed for workers”.

• Doc (Randomness note): “DataLoader will reseed workers following the multi-process algorithm”.

• Our observation: In Case C, no reseed happened on epoch boundary (we saw continuous RNG). This contradicts an intuitive reading of “reseed” unless one expects worker_init_fn.

If discrepancies arise, we define repair boundaries: what exactly is broken, and how much fix is needed. For example, if persistent reuse means no new base_seed, one repair could be to force a new generator call on each epoch. We document these potential fixes here without applying them yet.

Define Repair Boundaries

We measure how far observed behavior is from ideal. For example, “persistent=True without shuffle leads to stale RNG after epoch 0.” This is a binary condition: PASS (the bug is present) or FAIL (behavior is as desired).

We note also any contradictions: e.g., if the reproducibility docs suggest reseeding but the actual code does not, we honestly record that inconsistency.

11. Implement a Bounded Repair

If evidence shows a problem, we propose minimal changes. A common fix is to explicitly reseed in the fixture or to clear workers each epoch. Since we cannot change PyTorch’s internals in a release, we focus on workarounds. For instance, if persistent workers aren’t reseeded, one repair is to close and recreate the DataLoader each epoch (effectively making workers non-persistent). Or we could call torch.manual_seed() again in code. Example patch (pseudocode):

# Workaround: manually reseed generator each epoch when using persistent_workers
for epoch in range(2):
    # (Explicit reseeding step before iterating)
    g.manual_seed(100 + epoch)  # or reinstantiate DataLoader each epoch
    dl = DataLoader(
        dataset, shuffle=True, num_workers=2,
        persistent_workers=True, generator=g, worker_init_fn=seed_worker
    )
    for batch in dl:
        process(batch)

We treat this as an "optional fix" path. We include code snippets EXPECTED to illustrate it. We do not claim these were run, but we reason their effect. The key is that this fix must be bounded: it solves the specific test issue without broad side effects. For example, constantly recreating workers has overhead, but if correctness demands it, that’s the cost.

If persistent behavior is allowed but needs coordination (e.g. calling dl._iterator._reset(dl) manually), we mention that. We could also propose contributing a patch to PyTorch’s code if warranted. But for now, we focus on in-place fixes. The goal is to ensure new epochs truly get new randomness.

12. Verify Rollback and Rerun Safety

Any fix or configuration change must be reversible and testable. For example, if we decide to disable persistent_workers (ROLLBACK from persistent to default), we should ensure that running the tests again yields the expected baseline behavior. Conversely, if we apply our “repair” (like manual reseeding), we rerun the relevant cases to see if the issue is resolved. All these runs and their outputs belong in our evidence ledger.

We also check that failing cases remain failing if we undo the fix (to ensure we actually fixed the right thing). For rollback: if a colleague reverts our modifications, the original issue should reappear in tests (confirming that fix was indeed responsible for correctness).

Assign Operational Ownership

Finally, we decide who maintains this test or fix. For instance, “Make the ML engineering team responsible for including this fixture in CI.” The pass condition: an explicit owner is named (e.g. “Data Engineering Team”). No owner = FAIL.

13. Decide ACCEPT, REPAIR, RERUN or HOLD

Based on the evidence, we fill a decision ledger. The example below is EXPECTED / NOT EXECUTED:

Given our synthetic fixture, we likely find that sampler randomness is fine (since the shared generator covers it). If so, sampler assertion is ACCEPT. If worker-side RNG does not reset by default, that’s probably a REPAIR: either mandate worker_init_fn usage or disable persistent. If everything matched documentation, we ACCEPT; if we found a bug, perhaps HOLD (await fix) or REPAIR via workaround.

We then have a clear release decision: e.g. ACCEPT with note that users should use worker_init_fn. We document this in prose. For example:

Decision: ACCEPT the DataLoader behavior as expected, with the stipulation that reproducibility requires using torch.Generator and worker_init_fn per PyTorch recommendations. We do not find a defect in the code that mandates a library fix, but we emphasize the documented approach. If anything, we recommend adding an example of this in the DataLoader documentation (REPAIR to docs).

If we had found a PyTorch bug, we would choose REPAIR (e.g. “file a bug and patch code”). RERUN might be chosen if tests were non-conclusive (e.g. flaky outcomes) and need retesting. HOLD could mean “issue won’t be resolved now, but to be re-checked later.” Each option is justified in the ledger row.

14. Connect the Practice to Refonte Learning

This exercise exemplifies the rigorous, evidence-based approach taught in Refonte Learning’s Data Science & AI program. The program emphasizes not just machine learning theory but practical engineering discipline: isolating variables, writing explicit acceptance criteria, and verifying behavior with controlled experiments. In fact, Refonte’s curriculum covers topics like Python programming, statistical modeling, and deep learning, all through hands-on projects. This playbook is a data-engineering analog: it uses tools (Python, PyTorch) and project-based testing to ensure correctness before deployment. Such skills, designing reproducible fixtures, analyzing framework documentation, and making clear release decisions, are exactly the kinds of competencies the Refonte Data Science & AI program develops.

If you’re interested in structured training on these topics, see Refonte Learning’s Data Science & AI program, which covers Python data science, statistical modeling, machine learning, deep learning, generative AI, and practical project application. The program (3 months, 12–14 hours/week) includes mentorship, projects, and a certificate, providing a pathway to apply such rigorous methods professionally.a