Saving itertools.groupby Groups Does Not Save Their Rows

Executive summary. A Python data pipeline can appear to save three groups successfully and still retain zero rows. The failure in this fixture is an ownership problem: each subgroup returned by itertools.groupby() shares the same underlying input as the outer iterator. Saving a subgroup handle does not preserve an independent copy of its rows. Python's groupby() documentation explains that groups cover consecutive equal keys and that advancing the outer iterator makes the previous group unavailable. Store the group data while it is current if a later consumer needs it.

This playbook uses one finite, immutable five-row fixture and an independent membership oracle. The source rows are A1/A/10, A2/A/20, B1/B/30, B2/B/40, and C1/C/50; the expected memberships are A=[A1,A2], B=[B1,B2], C=[C1]. These expected memberships are declared as literals in the fixture. They are not derived from whatever the grouping program happens to retain.

The operational rule is simple but strict: ACCEPT only when later-readable rows match the declared five-row population and exact memberships. REPAIR by consuming or materializing each group before outer advancement, or by choosing an explicitly global grouping algorithm. HOLD incomplete retention, violated adjacency, empty input when five rows are required, or missing provenance. REOPEN_AND_RERUN only from retained originals, never from the already-spent source cursor.

Declare the retained-row contract

The first mistake when investigating groups that are empty later is to count stored pairs instead of retained rows. Saving three objects shaped like ("A", group_iterator), ("B", group_iterator), and ("C", group_iterator) proves only that three consecutive-key runs were observed. It does not prove that any downstream consumer can still read A1, A2, B1, B2, or C1 from those subgroup handles.

For this fixture, the contract is independent and literal:

Every main key is a present, non-null string, and the main source is already consecutive by key. That boundary matters. Missing-key handling in pandas GroupBy is a separate policy question about which rows enter the grouped result. The main experiment here removes that issue from consideration so that the lifetime of borrowed subgroup iterators is the only retention defect under test.

A group handle is not an owned row snapshot

Define success in terms of what can be read later, after control has moved past the group. An owned tuple such as ("A", (("A1","A",10), ("A2","A",20))) carries row references forward. A borrowed subgroup iterator does not promise an independent replay. The acceptance test therefore compares exact row identifiers, multiplicity, and membership against the oracle; the number of keys alone is insufficient. Here, row identity means the fixture's row_id value, not Python's id() or is object-identity test.

A useful contract record also separates four categories of evidence. The Python documentation defines the library behavior. The literal EXPECTED mapping defines the authored expectation. The probe output records an observation from one runtime. The ACCEPT/REPAIR/HOLD/REOPEN_AND_RERUN labels are operational policy layered on top. Mixing those categories creates weak audits: a passing assertion does not rewrite the specification, and documentation describing iterator lifetime does not prove that a particular business manifest is correct. Keeping them separate makes later reviews much easier.

Explain the borrowed subgroup lifetime

Python documents two properties that must be read together. First, groupby() returns consecutive keys and groups, starting a new group whenever the key value changes. Second, the subgroup iterator shares the underlying iterable with the outer groupby() object; advancing the outer iterator makes the previous group unavailable. The documentation therefore recommends storing group data if it is needed later.

That shared-input design creates a precise ownership boundary. Constructing groupby(source, key=key) does not consume the full source. Asking the outer iterator for a (key, group) pair identifies the current run. Consuming a subgroup advances through that run and can read and buffer the first row of the next run. If the current subgroup is unfinished, requesting the next outer pair skips its remaining rows to find the next key; an already-buffered boundary can be reused. Python's documented rough equivalent shows both the boundary lookahead and the draining of an unfinished group.

The diagram shows one source cursor shared by the outer iterator and its subgroup handles. Each handle provides a temporary view of a consecutive run. Saving those views does not save their rows. For this fixed fixture, the loss follows from cursor advancement and can be reproduced without any timing or garbage-collection race.

This distinction is especially important at API boundaries. A function that yields (key, group) pairs may look like it is returning ordinary values, yet the group component has a lifetime coupled to iteration state outside that object. Returning those pairs from one layer and consuming them in another silently transfers a cursor-lifetime obligation. If the receiving layer cannot guarantee immediate consumption, the producing layer should usually materialize the subgroup or change the interface so ownership is explicit.

Build the immutable source and yield trace

The fixture is intentionally small enough to audit by eye and strict enough to detect the defect. Rows are immutable tuples of (row_id, group_key, amount), and each experiment starts from a fresh iterator. The Tracked wrapper records which row IDs the source yielded, while the retention checks record which IDs remain available to the eventual consumer. Those are different ledgers.

The probe uses only the Python standard library, a five-row synthetic input owned by the fixture, literal assertions, and a fresh Tracked(ROWS) object whenever source position matters. No test derives its expected membership from the output under test. The negative controls are expected to produce rejected or HOLD states; their presence does not make the overall probe a failure. A green harness means all declared outcomes matched, including the deliberately defective ones. It does not mean every case satisfies the retained-row contract.

The complete driver below was executed with CPython 3.12.14 and returned EXPECTED_OUTCOMES_MATCH for all nine declared cases. That observation is limited to this interpreter and finite synthetic fixture. Save the listing as groupby_probe.py and run python3 groupby_probe.py with a matching interpreter to reproduce the reported environment. The driver rejects optimized execution because its checks use assertions; Python's assert statement documentation explains why optimization removes those checks.

The execution record has a limited evidentiary scope. It shows that this finite driver, with these assertions and immutable rows, behaved as reported on the named interpreter. It does not benchmark performance, measure memory use, validate alternate Python implementations, or show that a remote production source can be replayed. The JSON records each case's observed retained data and policy disposition. The literal EXPECTED mapping remains the independent membership oracle. Observations are checked against expectations; a repeatable defective output does not become correct merely because it is repeatable.

The test is intentionally small enough that a reviewer can reconcile every identity manually without trusting another grouping implementation.

"""Keep group membership evidence separate from borrowed group iterators."""
from collections import Counter, defaultdict
from itertools import groupby
import json
import sys
if not debug:
    raise RuntimeError(
        "Run without -O or PYTHONOPTIMIZE; assertions are required."
    )
ROWS = (("A1","A",10),("A2","A",20),("B1","B",30),
        ("B2","B",40),("C1","C",50))
EXPECTED = {"A": ["A1","A2"], "B": ["B1","B2"], "C": ["C1"]}
EXPECTED_IDS = ["A1","A2","B1","B2","C1"]
def key(row):
    return row[1]
class Tracked:
    def init(self, rows):
        self.source = iter(rows)
        self.yielded = []
    def iter(self):
        return self
    def next(self):
        row = next(self.source)
        self.yielded.append(row[0])
        return row
def retain(rows):
    return [(k, tuple(g)) for k, g in groupby(rows, key=key)]
def identities(groups):
    return [(k, [row[0] for row in group]) for k, group in groups]
def main():
    records = []
    tracked = Tracked(ROWS)
    borrowed = list(groupby(tracked, key=key))
    late = identities([(k, tuple(g)) for k, g in borrowed])
    assert late == [("A",[]),("B",[]),("C",[])]
    assert tracked.yielded == EXPECTED_IDS
    records.append({
        "case":"deferred_groups",
        "retained":late,
        "source_yielded":tracked.yielded,
        "decision":"HOLD_REPAIR"
    })
    retained = retain(Tracked(ROWS))
    assert identities(retained) == list(EXPECTED.items())
    records.append({
        "case":"immediate_materialization",
        "retained":identities(retained),
        "decision":"ACCEPT_CANDIDATE"
    })
    tracked = Tracked(ROWS)
    outer = groupby(tracked, key=key)
    first_key, first = next(outer)
    first_row = next(first)
    second_key, second = next(outer)
    assert (first_key, second_key, first_row[0]) == ("A","B","A1")
    assert tuple(first) == ()
    remaining = [(second_key, tuple(second))] + [
        (k,tuple(g)) for k,g in outer
    ]
    retained_ids = [first_row[0]] + [
        r[0] for ,group in remaining for r in group
    ]
    assert retainedids == ["A1","B1","B2","C1"]
    assert tracked.yielded == EXPECTED_IDS
    missing = Counter(EXPECTED_IDS) - Counter(retained_ids)
    missing_ids = list(missing.elements())
    assert missing_ids == ["A2"]
    records.append({
        "case":"partial_group_then_advance",
        "retained_ids":retained_ids,
        "missing_ids":missing_ids,
        "source_yielded":tracked.yielded,
        "decision":"HOLD"
    })
    aggregates = {}
    for k, group in groupby(Tracked(ROWS), key=key):
        ids, total = [], 0
        for row in group:
            ids.append(row[0])
            total += row[2]
        assert k not in aggregates
        aggregates[k] = {
            "ids":ids,
            "count":len(ids),
            "sum":total
        }
    assert aggregates == {
        "A":{"ids":["A1","A2"],"count":2,"sum":30},
        "B":{"ids":["B1","B2"],"count":2,"sum":70},
        "C":{"ids":["C1"],"count":1,"sum":50}
    }
    records.append({
        "case":"consume_aggregate_immediately",
        "groups":aggregates,
        "decision":"ACCEPT_AGGREGATE_CONTRACT"
    })
    interleaved = (ROWS[0],ROWS[2],ROWS[1],ROWS[3],ROWS[4])
    fragments = identities(retain(interleaved))
    assert fragments == [
        ("A",["A1"]), ("B",["B1"]), ("A",["A2"]),
        ("B",["B2"]), ("C",["C1"])
    ]
    assert dict(fragments) == {
        "A":["A2"],
        "B":["B2"],
        "C":["C1"]
    }
    records.append({
        "case":"nonconsecutive_keys",
        "fragments":fragments,
        "decision":"HOLD_FOR_CONSECUTIVE_GROUP_CONTRACT"
    })
    sorted_groups = retain(sorted(interleaved, key=key))
    assert identities(sorted_groups) == list(EXPECTED.items())
    records.append({
        "case":"sort_then_retain",
        "groups":identities(sorted_groups),
        "decision":"ACCEPT_CANDIDATE"
    })

    global_groups = defaultdict(list)
    for row in interleaved:
        global_groups[key(row)].append(row)
    global_ids = {
        k:[r[0] for r in v]
        for k,v in global_groups.items()
    }
    assert global_ids == EXPECTED
    records.append({
        "case":"global_dictionary_grouping",
        "ids":global_ids,
        "decision":"ACCEPT_CANDIDATE"
    })
    empty = retain(())
    assert empty == []
    records.append({
        "case":"empty_source",
        "retained":empty,
        "decision":"HOLD_EXPECTED_FIVE_ROWS"
    })
    single = list(groupby(ROWS[:2], key=key))
    single_late = [(k,list(g)) for k,g in single]
    assert single_late == [("A",[])]
    records.append({
        "case":"single_saved_group",
        "retained":single_late,
        "decision":"HOLD_REPAIR"
    })
    print(json.dumps({
        "python":sys.version,
        "cases":records,
        "status":"EXPECTED_OUTCOMES_MATCH"
    }, indent=2))
if name == "__main__":
    main()

A source can yield a row that the consumer never retains

The Tracked.yielded list is evidence of source traversal, not evidence of retained output. In deferred_groups, the source yield ledger contains all five IDs while the later-readable groups contain none. In partial_group_then_advance, A2 appears in the source ledger but disappears from the retained manifest. The source did its job; the retention boundary failed. A robust audit therefore keeps both ledgers rather than treating “seen by the wrapper” and “owned by downstream” as synonyms.

Save the outer pairs and inspect the empty groups

The misleading line is compact:

borrowed = list(groupby(tracked, key=key))

It looks like eager retention because list(...) is eager. But the list is eager only with respect to the outer iterator. Its elements are (key, subgroup_iterator) pairs; it does not recursively materialize each subgroup. To discover A, then B, then C, and finally know that no fourth group exists, the outer iterator advances the shared source. By the time list(groupby(...)) finishes, the saved subgroup handles no longer own unread rows. The later result is exactly:

A=[]
B=[]
C=[]
source_yielded=[A1,A2,B1,B2,C1]

That observation matches the documented shared-source contract: earlier group data is unavailable after outer advancement unless it was stored. The keys were found and the source yielded its rows, but the saved objects were borrowed iterators rather than row snapshots.

A common debugging misstep is to add logging around the source, see all five IDs pass through, and conclude that the downstream stage must have received them. This fixture shows why that inference is invalid. The source trace answers “what did this cursor yield while groupby advanced?” The retained manifest answers “what data can the later consumer still enumerate?” Both can be true simultaneously: every source row was traversed, while zero rows were preserved in the saved subgroup handles.

The related zip(strict=True) consumption playbook examines a different, two-feed contract: it checks paired lengths and identities while consuming both iterators. This example has one feed. The shared operational lesson is that consumption boundaries matter, and wrapping an advanced iterator in a new consumer does not rewind it.

Advancing the outer iterator changes the shared cursor

Do not diagnose this as subgroup objects expiring because they sat too long. In this fixture, elapsed wall-clock time is irrelevant. The decisive event is advancement of the outer iterator over a shared source. A saved subgroup does not contain a starting offset plus a private cursor that can be replayed later. Its ability to yield rows depends on where the shared groupby machinery has moved the source.

Advance after partially consuming a group

The all-empty case is dramatic, but the partial case is more operationally important because it resembles a real consumer that samples a row and then defers the rest.

The driver first obtains group A, consumes only A1, and then asks the outer iterator for the next group:

first_key, first = next(outer)
first_row = next(first)            # A1
second_key, second = next(outer)  # outer advances to B

To reach key B, the groupby machinery must move past the remaining A row. A2 is therefore consumed from the shared source while finding the boundary, but it was never copied into an owned output structure. A subsequent tuple(first) is empty. The final retained IDs are A1, B1, B2, C1; A2 is missing even though the source-yield trace contains all five IDs.

This is a stronger negative control than the deferred-all-empty case. It shows that row loss can happen before the outer iterator is exhausted and can affect only an unretained suffix of one group. In a larger pipeline, that can produce plausible partial output rather than a conspicuous set of empty groups.

The acceptance decision is therefore HOLD, not “close enough because group A was observed.” Exact identity reconciliation fails: the expected population has five IDs, while the retained population has four. The correct repair is to consume or materialize the rest of A before requesting B, or to redesign the consumer so no downstream stage expects raw A rows after the cursor moves.

Partial consumption also rules out a simplistic “just keep the first row” workaround. Once a consumer decides that a group-level object may outlive the current outer-loop iteration, it needs a complete retention rule for that group. Sampling one row, computing one field, or checking one predicate does not preserve the unconsumed suffix. The acceptance boundary belongs at the point where the group is either fully consumed or explicitly abandoned under a contract that says the remaining rows are irrelevant.

Materialize each group at the consumption boundary

The direct repair is to convert each subgroup into an owned finite container inside the outer loop, before the next group is requested:

def retain(rows):
    return [(k, tuple(g)) for k, g in groupby(rows, key=key)]

Here tuple(g) drains the current subgroup while it is still valid. Only after that tuple is complete does the comprehension advance the outer groupby iterator. On the five-row source, the retained result is A=[A1,A2], B=[B1,B2], C=[C1], exactly matching the independent oracle. That makes it a candidate for ACCEPT, assuming provenance and the five-row population contract are also present.

This change is small syntactically but significant architecturally. The handoff now carries an owned finite sequence of row references instead of an iterator whose lifetime depends on an upstream cursor. Python's tuple() constructor consumes an iterable and stores the yielded items in order, which creates a durable container for this finite case.

Memory policy must be explicit. Retaining every materialized group requires O(N) row references plus container overhead, where N is the retained row count. Processing and discarding one materialized group at a time uses O(Gmax) additional group-reference storage, where Gmax is the largest group. That bound excludes retained source data, accumulated output, downstream state, and audit evidence. The fixture itself retains all originals in ROWS. A ledger containing every row ID also grows with the population, so processing one group at a time does not make the complete workflow constant-memory.

For finite feeds, this is a policy choice rather than a universal optimization target. If the downstream step needs all raw rows after grouping, O(N) retained references are expected somewhere. If downstream can finish each group before moving on, materializing one group at a time can isolate the lifetime hazard without retaining the whole result. What should be rejected is an accidental memory policy in which borrowed iterators are kept because they appear cheap, even though the consumer actually requires durable row access later.

Tuple retention is not arbitrary deep copying

A tuple stores references to its elements; it does not recursively freeze arbitrary nested payloads. In this fixture the row tuples contain only immutable strings and integers, so later mutation of those row values is not a concern. If a row contained a mutable list, dictionary, or custom object, tuple materialization would retain a reference to that object rather than deep-copying its state. Document that limitation separately from the iterator-lifetime contract.

Aggregate immediately under a different output contract

Sometimes downstream consumers do not need raw rows at all. In that case, a valid design is to consume each subgroup immediately and emit only the approved aggregate. The driver computes an ID ledger, row count, and sum while the group is current:

A: ids=[A1,A2], count=2, sum=30
B: ids=[B1,B2], count=2, sum=70
C: ids=[C1],    count=1, sum=50

This satisfies a declared aggregate contract because every subgroup is fully consumed before the outer iterator advances, and the retained ID ledger allows exact membership reconciliation against the authored oracle. The amounts also match the expected 30, 70, and 50.

But do not silently promote an aggregate-only interface into a raw-row retention interface. If the consumer later asks for every original row, counts and sums cannot reconstruct those rows. Even an exact total of 150 does not prove that all five intended identities were present; compensating errors can preserve totals. The ID ledger is what makes this particular fixture auditable.

The memory implication is equally important. An aggregate implementation that keeps only count and sum can use very little state per group, but the supplied case intentionally retains every row ID as evidence, so its audit storage still scales with the number of rows. A production design that wants O(number of groups) aggregate memory needs a separately approved provenance policy, such as immutable input manifests or durable row-level evidence stored elsewhere. The algorithm cannot obtain a smaller memory bound merely by calling its output “an aggregate.”

The aggregate contract should therefore state its evidence granularity. For this fixture, ACCEPT requires exact IDs as well as count and sum. Another production contract might retain a content hash, source object version, partition manifest, or durable row-level lineage outside the process. That can be valid, but it is a different provenance design. The important point is to decide before processing which evidence downstream review requires, then consume the subgroup completely while that evidence can still be derived.

Expose the separate consecutive-key requirement

Fixing subgroup lifetime does not make itertools.groupby() a global grouping operator. Python specifies consecutive grouping: a new group starts whenever the key changes. The control input makes that visible by interleaving the same five rows as A1, B1, A2, B2, C1.

Immediate materialization is now perfectly correct with respect to iterator lifetime, yet the runs are:

A -> [A1]
B -> [B1]
A -> [A2]
B -> [B2]
C -> [C1]

That output should be HOLD_FOR_CONSECUTIVE_GROUP_CONTRACT if the declared requirement is one complete A group, one complete B group, and one complete C group. The defect is no longer borrowed lifetime; it is violated adjacency.

A second trap appears when code converts those fragments directly to a dictionary:

fragments = [
    ("A", ["A1"]),
    ("B", ["B1"]),
    ("A", ["A2"]),
    ("B", ["B2"]),
    ("C", ["C1"]),
]
dict(fragments)
# {'A': ['A2'], 'B': ['B2'], 'C': ['C1']}

Python's dictionary constructor retains the last supplied value for a repeated key, so A1 and B1 disappear from that representation. The fragments were already materialized. The loss here comes from replacing earlier dictionary values, independently of subgroup lifetime.

This separation prevents a dangerous repair cascade. A team can correctly notice empty deferred groups, add tuple(g), see rows reappear, and still ship incomplete global groups because repeated keys were never meant to be adjacent. The acceptance harness must test both dimensions independently: subgroup lifetime and key adjacency. Passing the first does not waive the second. In practice, the input contract should state whether ordering by group key is guaranteed, checked, or intentionally irrelevant because a global algorithm is used.

Consecutive grouping and global grouping are different operations

Choose explicitly: require adjacency as an input invariant, sort by the grouping key before using groupby(), or use a global accumulation structure. The standard-library documentation distinguishes consecutive and global grouping. Assuming global semantics after repairing subgroup lifetime can still produce incomplete groups.

Compare sorting with dictionary-based grouping

Two approved alternatives repair the interleaved-input case, but they encode different contracts.

Sort, then group. sorted(interleaved, key=key) creates a new sorted list. Python's stable sorting behavior preserves the relative order of rows with equal keys, so A1 remains before A2, and B1 before B2. Once equal keys are adjacent, immediate tuple materialization yields the expected complete A, B, and C groups. Sorting materializes the iterable into a new list and performs an ordering step; it changes cross-key order for this interleaved input. You must still consume each subgroup before advancing if its rows are needed later.

Accumulate globally. A defaultdict(list) can append each row to the list for its key as the source is scanned. Python's defaultdict examples use this list-factory pattern to group key-value pairs into a dictionary of lists. Equal keys need not be adjacent, so the interleaved fixture still produces A=[A1,A2], B=[B1,B2], C=[C1].

These alternatives are implementation choices inside a Python data pipeline. The broader SQL and Python for data engineering discussion explains where each language fits in a workflow. Here, the decision concerns how the Python stage groups rows and retains them for its next consumer.

Test empty input and a single saved group

Boundary cases are useful because they expose false acceptance rules.

For an empty source, retain(()) correctly returns an empty list. That behavior is not an application success when the declared population is five rows. Under this playbook's contract, the disposition is HOLD_EXPECTED_FIVE_ROWS. An empty result is only acceptable if the upstream population contract explicitly allows zero rows; here it does not. The absence of an exception is not evidence that the expected data arrived.

The one-key control is even more revealing:

single = list(groupby(ROWS[:2], key=key))
[(k, list(g)) for k, g in single]
# [('A', [])]

Only one key exists, yet the saved subgroup is still empty after outer exhaustion. That disproves a tempting explanation that the bug requires several groups overwriting one another. The issue is not “B stole A's data.” Asking list() to finish the outer groupby() must eventually determine that there is no next group, which exhausts the shared source. The saved A handle does not become a replayable two-row container merely because there was no B key.

These controls also sharpen acceptance criteria. Group count is weak evidence: deferred_groups reports three keys but zero retained rows; single_saved_group reports one key but zero retained rows; empty_source reports zero keys and zero rows. All three can look structurally plausible if validation checks only “did code run?” or “does the container have the expected type?” Identity-based reconciliation distinguishes them immediately.

The empty-source control is particularly important in scheduled batch work. “No exception” can mean either a legitimate zero-row batch or a broken upstream handoff. The program cannot infer which from iterator mechanics. Here the fixture author has declared five required rows, so zero is unambiguously HOLD. In another system, accepting zero would require an explicit population policy and provenance showing that the authoritative source really contained no eligible rows. The grouping code should not invent that business fact.

Apply identity-based acceptance criteria

A production acceptance gate should compare outputs to an independently declared manifest, not to the output's own self-description. For this finite fixture, the decisive evidence is exact row identity, multiplicity, group membership, and provenance.

The word “candidate” is deliberate. A row-perfect artifact with unknown provenance should still be HOLD. The same applies if duplicate IDs appear where uniqueness is required, if a key violates the consecutive-run assumption, or if the declared expected population is itself absent. Technical consistency cannot replace an input-ownership record.

For this ordered fixture, compare the complete sequence with identities(retained) == list(EXPECTED.items()). This checks the number and order of runs as well as each run's ID sequence, so an extra earlier group cannot disappear inside a dictionary conversion. A production acceptance function may instead compare the full multiset of retained IDs with the expected multiset, verify each ID's group key, reject repeated runs where required, and check group-specific measures. This fixture requires each ID exactly once. Amounts provide corroboration after membership has been reconciled; a correct sum can coexist with wrong membership.

The data-cleaning workflow recommends preserving raw inputs and defining the intended row grain before transformations. Those practices provide the source and data-quality evidence needed for a controlled repair. Here they become a concrete iterator-recovery requirement: retain an authoritative source from which a fresh cursor can be opened.

Missing retained rows require HOLD, not an invented fill

Do not synthesize A2 because A's expected total was 30, and do not “repair” a four-row manifest by fabricating a fifth identifier. Aggregate arithmetic cannot recover an exact lost row. Preserve the defective artifact, mark the missing identity, and reopen from an authoritative retained source. A HOLD disposition is evidence that the contract was not met, not a failure of the audit process.

Recover with new cursors over retained originals

Recovery starts from an object that still represents the authoritative input. In this fixture that object is the immutable ROWS tuple. It does not start by wrapping the already-spent Tracked instance again.

Python's iterator protocol requires an iterator's iter() to return the iterator object itself. Once next() raises StopIteration, subsequent calls must continue to do so. Creating another consumer around the same advanced cursor therefore does not rewind it. A new groupby(old_tracked, key=key) starts from whatever position old_tracked has reached.

A controlled rerun is:

reopened = Tracked(ROWS)  # fresh cursor over retained originals
retained = retain(reopened)  # materialize before outer advancement
assert identities(retained) == list(EXPECTED.items())

Operationally, preserve both artifacts. Keep the defective report and its source-yield/retained-ID evidence so reviewers can see why it was rejected. Publish the repaired result as a new artifact with its own runtime, command, input provenance, and manifest. That distinction matters because a new successful run does not retroactively make the old output complete.

The label REOPEN_AND_RERUN should therefore mean “open a fresh cursor over retained originals or another authoritative replayable source, then execute the corrected logic.” It should never mean “try the same exhausted iterator again.” In systems where the original upstream input was not retained and cannot be reproduced, recovery may be impossible; this local fixture cannot establish otherwise.

For auditability, record the rerun as a new observation: command, interpreter, input identifier or checksum, transformation version, retained manifest, and disposition. Do not overwrite the rejected artifact in place. Preserving the before-and-after evidence lets reviewers distinguish the discovered defect from the corrected result and prevents a later green run from obscuring what downstream consumers may previously have seen.

Assign iterator ownership and memory budgets

Reliable finite-feed processing needs an explicit owner for cursor advancement. The component that owns the outer groupby() loop should also own the rule for what happens to each subgroup before it advances. A downstream function should receive a borrowed subgroup only when its contract requires consumption to finish before the caller requests the next group.

A useful handoff specification names four things: the input is finite; keys are consecutive or preprocessing makes them so; the consumer either receives owned rows or consumes the borrowed group within scope; and the audit/memory budget is explicit. Review the design whenever a formerly eager consumer becomes deferred, queued, cached, returned from a function, or stored for later. Those changes can cross the subgroup-lifetime boundary without changing the grouping line itself.

Code review should therefore look beyond groupby() syntax. The risky question is whether the consumer's time horizon changed. Moving work into a callback, generator, task queue, cache, or returned closure can turn a previously immediate consumer into a deferred one. At that point the producer must either extend ownership by materializing data or redesign the interface so the cursor owner remains active until consumption finishes. This is an API-lifetime concern as much as an iterator concern.

Memory budgets should distinguish payload from evidence. Keeping all materialized row groups stores references in proportion to the retained population. Processing and discarding one tuple at a time uses additional group-reference storage proportional to the largest group, excluding the retained source, output, downstream state, and audit ledger. Sorting adds a materialized ordered sequence. defaultdict(list) intentionally keeps global row collections. Choose the algorithm whose ownership and retention semantics match the declared downstream requirement.

There is also an ordering question beyond memory. Stable sorting preserves relative order within equal keys but can change order across keys; it does so for this interleaved fixture. A dictionary of lists preserves encounter order inside each key without imposing a global key sort. If original cross-key order is business evidence, retain the original manifest as well. Algorithm selection therefore requires explicit decisions about adjacency, ownership, and ordering.

These decisions belong within broader data-engineering processing responsibilities, including ingestion, transformation, pipeline automation, monitoring, and data-quality checks. At the iterator boundary, make cursor ownership and evidence retention concrete design choices that another engineer can review.

Build reliable data workflows with Refonte Learning

Use itertools.groupby() according to its documented lifetime and consecutive-key contract. When downstream code needs rows after the outer iterator moves, consume or materialize the subgroup at that boundary. If keys are not adjacent, sort deliberately or use a global accumulation algorithm. If the declared five-row manifest does not reconcile, HOLD. If the source cursor has already advanced, REOPEN_AND_RERUN from retained originals. Python's shared-source behavior and iterator-exhaustion rules explain these boundaries; the nine-case CPython 3.12.14 driver demonstrates them on this finite synthetic fixture.

That evidence remains deliberately narrow. It does not establish correctness for arbitrary business rules, mutable nested payload snapshots, every Python implementation, unknown remote side effects, or data whose provenance was never retained. What it does establish is enough for the decision at hand: a saved (key, group_iterator) pair is not an owned row snapshot, and acceptance must be based on later-readable identities rather than the appearance of saved group handles.

The durable engineering habit is to ask “who owns these rows after this line executes?” before handing an iterator-derived object to another stage. When the answer is ambiguous, the data contract is ambiguous. Resolve that ambiguity at the boundary, while the finite source is still available, rather than after a downstream audit discovers that the keys survived but the rows did not.

Readers developing ingestion, ETL, transformation, warehousing, and batch or stream-processing workflows can review Refonte Learning's Data Engineering Program and its published curriculum. The programme page describes three months at 12–14 hours per week, data warehousing and ETL, Hadoop and Spark, pipeline design, ingestion, transformation, and mentor guidance. It advertises a Training Certificate and a Certificate of Internship upon successful completion. The published curriculum does not specifically list this itertools exercise or a CPython version.