sort -u Removed Different Rows Because Their Keys Matched

The command exits with status zero. The output contains one row per business key. A pipeline dashboard reports success, yet a distinct source record has disappeared. Adding -u to a field-key sort can produce exactly this result when the intended contract was to remove only repeated complete rows.

Diagnosing GNU sort -u key duplicates starts with a decision: should the stage remove identical lines, quarantine conflicting records sharing a key, or select an explicitly approved survivor? Those contracts produce different acceptable outputs. A smaller file cannot tell you which contract was implemented.

This playbook uses five synthetic tab-separated values (TSV) records, an independently declared four-row manifest, and ten diagnostic cases. The source below passed all ten cases on GNU coreutils 9.4 and CPython 3.12.14 with LC_ALL=C. Python launches processes and records evidence; GNU utilities perform the deduplication under test. The original preparation environment was reported as GNU coreutils 9.7 and CPython 3.13.5. That earlier provenance is separate from the reproduced run described here, and a manual edition must not be confused with either tested binary version.

The central observation is narrow: keyed uniqueness retained three rows, while complete-line uniqueness retained the expected four. Reordering the same input changed the surviving A record. A separate comm comparison returned zero while explicitly reporting the missing row.

The operational goal is therefore a reconciled dataset under a named contract. Preserve the source and failed candidate, explain every exclusion, and authorize downstream use only after identity and content checks pass.

Choose the equality relation before removing any row

Write down what two interchangeable records look like before editing the command. For this laboratory, exact-line equality means the same three ASCII fields with the same tab separators. All records have normalized newline termination. Under that contract, repeated occurrences of one complete row may collapse, but another record with the same business key must survive if its identifier or payload differs.

Business-key equality asks a different question. Here the first field is the key. Two A records belong to the same key group even when their remaining fields disagree. Finding such a group establishes a collision for a one-row-per-key destination; it does not establish which candidate is correct.

A reviewed survivor policy adds the missing authority. It specifies which source identity may represent each key and records the approval behind that choice. The rule must exist independently of whichever row a utility happens to emit.

These are release decisions, not interchangeable spellings of “deduplicate.” A downstream requirement for one row per key cannot quietly replace an upstream promise to retain every distinct record.

Related data-cleaning decision rules provide broader context for defining row meaning: that page distinguishes exact duplicates from repeated business events and emphasizes establishing row grain before deletion. Here, make the decision concrete enough to test against literal bytes rather than relying on the word “duplicate.”

Record the contract beside the stage configuration. Include the expected population, permitted exclusions, decision owner and empty-input policy. If nobody can authorize the loss of one distinct A record, keyed uniqueness is not an acceptable shortcut.

Freeze a five-row TSV input and a four-row oracle

The input is closed and deliberately small. Each record contains exactly three ASCII fields: business_key, record_id and payload. The separator is one literal tab, and encode appends one newline to every record. There is no header, quoting convention, embedded delimiter or embedded newline.

In original order, the file contains these five records. The fields below are separated by literal tabs; the Python byte literals in the complete driver define the authoritative fixture if copying from Word changes whitespace or line endings:

B             r3           blue
A             r2           green
A             r1           red
A             r2           green
C             r4           gold

The second and fourth occurrences are identical. The third occurrence shares their A key but differs in both record ID and payload. B and C each have one record. All keys are present; this experiment does not exercise missing-key behavior.

Before running any candidate transformation, declare the exact-row oracle independently:

A             r1           red
A             r2           green
B             r3           blue
C             r4           gold

The oracle is a reviewed expectation, not output copied from the command being evaluated. Its four records follow directly from the chosen contract: remove one repeated occurrence of A/r2/green, retain the other occurrence, and retain every other distinct row. A test that calculates its expectation by executing the candidate again would only confirm self-consistency.

The literal encoding makes the byte accounting inspectable. The original input occupies 51 bytes, including its five terminating newlines. The exact oracle occupies 40 bytes. The authorized reduction is therefore the 11-byte repeated r2 occurrence. These sizes describe this fixture only; they measure neither throughput nor the impact of a production incident.

Use LC_ALL=C and LANG=C for every subprocess. The purpose is a fixed comparison environment, not a claim about other locales. Likewise, this fixture defines exact-row identity within a normalized line format. It does not establish arbitrary byte-stream identity for files with different terminators or malformed records.

This protocol is deliberately not a general CSV parser experiment. It accepts only the declared normalized TSV format and uses line-oriented GNU utilities in that bounded setting. GNU sort and uniq document text-line comparison behavior, not business-entity identity. Treat a quoted delimiter or malformed field structure as a different input-contract problem instead of improvising another option during this test.

Keep source identity separate from the business key

The first field groups records; the second identifies the candidate record in this fixture. A/r1/red and A/r2/green are therefore two distinguishable candidates under one business key. Their shared key supplies no evidence that one supersedes the other.

The identifiers also make loss visible. A row count can report that three records remain, but only an identity comparison reveals whether r1 or r2 disappeared. The payload provides another check: retaining an ID while changing its associated value would still violate this oracle.

Do not infer an update sequence from the digits in r1 and r2. They are labels, not timestamps or priorities. Red and green are equally synthetic payloads. Neither lexical order nor the order of arrival confers authority.

Preserve the original occurrence order as evidence even though the exact-row output is sorted. The experiment later changes only that order. Keeping the original ledger allows a reviewer to verify that the second run reused the same population rather than substituting a different record.

Run the keyed unique command and retain its output

Start by reproducing the suspect transformation against the frozen fixture. The important arguments are the tab separator, the bounded first-field key and -u. In a shell, with input.tsv containing the original bytes, the equivalent command is:

export LC_ALL=C LANG=C
tab=$(printf '\t')
sort -t "$tab" -k1,1 -u < input.tsv > keyed-candidate.tsv

The laboratory supplies the same bytes through standard input and passes the delimiter as a literal tab in an argument list. It captures the return code, standard output and standard error before asserting the observed result.

The recorded run returned zero and produced:

A             r2           green
B             r3           blue
C             r4           gold

That is the expected outcome of case S1. It is not the approved exact-row output. The candidate contains three rows and three keys, but it omits the distinct A/r1/red record. Of the two removed input occurrences, only one is the redundant repetition authorized by the exact-row contract.

Its 31 output bytes expose the same discrepancy at another level. The reduction from 51 bytes combines the authorized 11-byte duplicate removal with the unauthorized loss of the nine-byte r1 record. Retain the missing record’s identity alongside that arithmetic; a byte total by itself cannot explain which content disappeared.

Retain this candidate with its command evidence. Replacing it immediately with a repaired output would make it harder to establish what the previous stage actually published. Give the failed candidate a separate identity and attach the contract it failed.

Distinguish three statements in the review: the process completed successfully; the diagnostic reproduced its expected behavior; the output satisfies the delivery contract. In S1, the first two statements are true and the third is false for exact-row deduplication.

This distinction also prevents a misleading regression report. The S1 test should continue to pass when it observes the unsafe three-row result, because its purpose is to expose the comparator mismatch. Approval belongs to a separate assessment of candidate content against the selected policy.

If your observed output differs, preserve it and investigate the recorded arguments, input bytes and binary version. Do not edit KEYED to make an unexpected result green. A divergence is evidence to explain before relying on the proposed repair.

Explain which comparison -u actually changes

The GNU sort manual describes key comparison followed, ordinarily, by a whole-line last-resort comparison when the keys tie. Both -u and -s disable that fallback. With -u, uniqueness is evaluated using the resulting active comparison. GNU also explicitly warns that the equivalence between sort -u and sort | uniq does not extend to arbitrary sort options.

For this fixture, -k1,1 limits the explicit key to field one. Every A record therefore has the same selected key. In the ordinary nonstable key sort, the remaining whole-line comparison distinguishes A/r1/red from A/r2/green. Once -u disables that fallback, those differing fields no longer rescue the records from the same uniqueness group.

That explains the data loss without inventing a parser failure. The utility can receive all five rows, understand the options and finish successfully while implementing an equality relation broader than the pipeline intended. The mismatch sits between the delivery contract and the configured comparator.

Keep the ending field in the command. GNU’s key syntax allows a key with no explicit end position to extend through the end of the line, so -k1 is not the requested first-field-only key. Substituting it would change the experiment and could conceal the original mistake.

When no explicit key is supplied, GNU documents the default key as the entire line. That is why plain sort -u is the relevant candidate for the exact-row policy in this narrow fixture, rather than another first-field keyed command.

For this normalized input, the important identity is the specified complete record content. Do not generalize the experiment to every file serialization, newline convention or option combination. The reproduced result is local evidence under GNU coreutils 9.4 and the C locale.

A stable survivor is not an authorized survivor

Suppose the same command repeatedly retains r2 from the original input. Reproducibility helps explain what happened, but the business question remains unanswered: why should r2 represent A? The fixture provides no authority, timestamp, revision sequence or quality ranking supporting that choice.

Stability is an ordering property. Approval is a decision about the meaning of the delivered record. An ordering rule can be repeatable and still be inappropriate for the dataset being published. Adding an explicit sort direction does not fill that gap.

Require a reviewer to explain the survivor using the declared policy and retained source evidence. “It appeared first” is only an observation about this input unless an independently approved contract gives that position meaning. This article makes no such contract.

The same discipline applies when a result looks plausible. A contains a green value in one candidate and a red value in another; both are syntactically valid. The available fields cannot establish which value is authoritative. Preserve the alternatives until a responsible owner supplies the missing decision.

Compare full-line uniqueness with sort followed by uniq

Case S2 evaluates the direct implementation of the exact-row contract: sort -u with no explicit field key or comparison modifier. Case S3 retains the specified nonstable key sort and applies ordinary uniq afterward:

sort -u < input.tsv > exact-candidate.tsv
sort -t "$tab" -k1,1 < input.tsv | uniq > composed-candidate.tsv

In the recorded run, both candidates matched the independently declared four-row oracle, including content and order. These are two tested paths to the same result for this fixture. They are not evidence that every pipeline containing sort and uniq has equivalent behavior.

The ordinary key sort in S3 produces the A rows in the order r1, r2, r2. The repeated complete r2 rows are now neighbors, so the following utility can remove the redundant occurrence without discarding r1. B and C follow, unchanged.

The GNU uniq manual defines ordinary behavior in terms of adjacent repeated lines: by default, all but the first line in each adjacent repeated run are discarded. Its comparison controls can change what is examined. This experiment uses uniq without those controls, so the complete row remains relevant.

GNU’s sort documentation explicitly limits the equivalence between plain sort -u and sort | uniq; arbitrary sort options cannot be substituted. Treat the precise argument vectors as part of the experiment, especially when a review proposes folding two stages into one.

For an exact-row stage that needs no separate business-key ordering, the full-line command expresses the chosen contract directly. If another stage genuinely requires the composed form, preserve the tested options and document the dependency on how the intermediate records are arranged.

The laboratory does not rely on a shell pipeline’s final status. It runs sort, records and checks that process, then passes its captured bytes to uniq and checks the second process independently. This makes a failed first stage visible instead of allowing a successful final process to stand in for the entire composition.

The shell snippets illustrate the transformations, while the complete driver below supplies their process-level evidence checks. Preserve that distinction when translating the example into an orchestrated job with separate stage statuses.

When evaluating a replacement, inspect the full output even if its count is four. A candidate that duplicates r1 and loses r2 could still have the desired length. Acceptance requires the expected identities and payloads, not merely a successful command or a reassuring row count.

Expose the stable-sort adjacency trap

Case S4 adds -s to the field-key sort before ordinary uniq. This looks like a modest ordering change, but the observed output no longer matches the exact-row contract:

sort -t "$tab" -k1,1 -s < input.tsv | uniq > stable-candidate.tsv

GNU documents -s as disabling the last-resort comparison so that lines whose specified keys compare equal retain their original relative order. In this fixture, the A occurrences are r2, r1, r2, so the stable key sort preserves that order. The resulting five-row output is:

A             r2           green
A             r1           red
A             r2           green
B             r3           blue
C             r4           gold

Walk through the intermediate sequence rather than treating uniq as a global inventory. The first r2 is followed by r1, so that boundary is not a repeated complete row. The r1 is followed by the second r2, which is also different. There is no adjacent pair of identical A rows for ordinary uniq to reduce. GNU’s documented adjacent-line behavior is exactly the constraint the negative control is exercising.

The output retains every source occurrence, including the duplicate that the exact-row policy intended to remove. It therefore fails for a different reason from keyed -u: the stable composition retains too much, while keyed uniqueness retains too little.

These two negative outcomes are useful together. They stop an acceptance check from treating “at least three rows” or “no distinct row missing” as sufficient. The exact-row output must contain every expected distinct record exactly once in the specified normalized representation.

The intermediate sequence is also the evidence needed to localize the mistake. If reviewers see only the final count, they may blame the wrong stage. Retaining both subprocess outputs shows that the stable sort supplied separated repeats and that the following uniq left that sequence unchanged.

Do not generalize one pipeline equivalence to another

S3 and S4 differ by one argument, yet their outputs differ by one retained occurrence. The successful S3 result cannot justify S4 simply because both command strings end with uniq. Stage composition depends on the properties delivered by the preceding stage.

Review ordering changes as semantic changes when another utility depends on adjacency. A request to preserve arrival order within a business key may be reasonable, but it creates a different requirement that must be reconciled with exact-row removal. Do not assume that retaining the old final command preserves the old behavior.

Keep S4 as a named regression case even after choosing S2 as the repair. It protects against a later refactor that reintroduces the same mistake through an apparently harmless stability flag. Its PASS means that the test successfully reproduced separated duplicates, not that five rows are suitable for publication under the four-row contract.

Change input order without changing the population

Case S5 moves A/r1/red to the front while preserving all five occurrences. The new source order is r1, r3, r2, r2, r4, with their original keys and payloads. No row is added, removed or edited. The exact-row oracle consequently remains the same four records.

Run the same first-field keyed unique command on this reordered input. The recorded result now contains A/r1/red, B/r3/blue and C/r4/gold. The candidate still has three records, but its A survivor has changed.

This is a controlled explanation of why counts are insufficient. Every count in the first three rows agrees across the runs. Only the record comparison exposes a different business result. An acceptance gate based on size or key coverage would approve both conflicting outputs.

Keep the reordered input hash with the second command record. A different byte-order hash is expected, because this test intentionally changes serialization order. The occurrence ledger demonstrates that the population and multiplicities remain unchanged. Together, those observations isolate input order as the changed variable.

This experiment tests one deliberate permutation, not every possible arrangement or implementation. Its evidential value is already sufficient for the narrow claim: the configured stage does not express the predeclared A-to-r1 approval, because the original-order run retained r2 instead.

If the reordered result happens to match a desired survivor mapping, do not approve the command on that basis. The test has shown that the match depends on the tested arrangement. The authorization must be encoded and reconciled independently of that arrangement.

Use the contrast during review: ask which input property entitles one candidate to survive. If the answer refers only to incidental order, either obtain approval for a genuinely meaningful ordering contract or retain the collision for resolution.

Quarantine conflicting keys or apply an approved mapping

Exact-row removal leaves both distinct A candidates. That is correct when the consumer accepts multiple records per business key. If the destination instead requires one record per key, the remaining A group needs a separate resolution step.

Case S7 inspects the four-row GNU output and reports A with both A/r1/red and A/r2/green. Python groups these retained rows to assert the collision population; it does not replace GNU as the deduplication engine. The case identifies what must be quarantined, but it does not implement a production exception queue.

A quarantine record should connect the affected key, complete candidates, original occurrence counts and source batch. Preserve both A candidates together so that an owner can evaluate the disagreement. B and C are unambiguous in this fixture; releasing them separately still requires an explicit policy permitting a partial batch.

For the approved-survivor branch, the laboratory states a synthetic owner decision in advance: A maps to r1, B maps to r3, and C maps to r4. It is not inferred from lexical order, from either keyed output, or from the colors.

S8 applies that mapping to the four-row exact population and asserts the three expected complete records: A/r1/red, B/r3/blue and C/r4/gold. The assertion checks identities and payloads as well as cardinality. Its short selection expression is a fixture-level policy demonstration, not a general approval system.

When applying this contract beyond the fixture, require exactly one matching candidate for every approved key and reject unknown keys or unresolved alternatives. A total of three selected rows is insufficient if the distribution across A, B and C is wrong. The fixed expected list in S8 supplies the identity check that its length assertion alone cannot provide.

Make collision resolution a separate signed-off decision

Assign the business owner responsibility for the mapping and the pipeline maintainer responsibility for implementing it faithfully. Capture the applicable batch or policy version, the chosen identity, the excluded alternatives and the approval reference. A first-or-last rule introduced only to make tests pass is not an approval record.

The related discussion of ambiguous source-row ownership addresses the same governance question in a different system: it separates a unique approved source from merely obtaining the expected mutation count. This laboratory performs no database writes and derives its utility behavior from GNU documentation and controlled evidence.

Reconcile the selected output against its own three-row approved manifest. Record r2 as an intentional exclusion under that policy. Do not compare it only with the four-row exact oracle and call every difference accidental loss; equally, do not retroactively approve r2’s loss just because a convenient command produced r1.

If a mapping is missing, selects an absent identity, or cannot distinguish multiple differing candidates, retain the conflict and HOLD release pending clarification.

Reconcile missing and extra complete records

Reconciliation answers two separate questions: which expected complete records are absent, and which delivered complete records are unauthorized? Both directions matter. A candidate can preserve the expected count by losing one record and adding another.

The GNU comm manual requires sorted inputs and defines three output columns for lines unique to the first input, unique to the second, and common to both. With --check-order, unsorted input is an error. GNU also states that normal completion returns zero independently of the comparison result.

In S6, expected.sorted contains the independent four-row manifest. The bad keyed output is sorted by complete line into actual.sorted. The comparison commands are:

comm --check-order -23 expected.sorted actual.sorted
comm --check-order -13 expected.sorted actual.sorted

Both run under the same C locale. The first reports A<TAB>r1<TAB>red; the second reports nothing. Both return zero. The missing-output content, rather than the process status alone, establishes that the candidate fails exact-row reconciliation.

For a repaired candidate, prepare compatible complete-line ordering under C and retain its original multiplicity. Do not silently apply another uniqueness reduction during audit preparation: that could hide the repeated occurrence exposed by S4. If the chosen contract requires each expected row once, check that requirement too.

Use --check-order as part of the audit. An ordering error is a failed comparison, not an empty difference result. Preserve the error and correct the comparison preparation before assessing membership.

The release gate must consume both difference outputs after checking both return codes. An empty missing-record file establishes only one direction; an empty extra-record file establishes the other. Record both findings against the same candidate version, so that a replacement between checks cannot combine evidence from two different artifacts.

The broader idea of group membership reconciliation is useful context because that article also treats membership evidence as stronger than an attractive aggregate. This comparison, however, operates on complete TSV records with present keys; there is no missing-value grouping experiment here.

Against the approved-survivor policy, substitute the separately approved three-row expectation. Keep the exclusion ledger linking the omitted r2 record to its approval. The comparison tool cannot decide which manifest is appropriate; that choice belongs to the contract review.

Finally, inspect the retained raw output representation. Most laboratory cases compare rows using splitlines(), while command records preserve the original ASCII output including newlines. The row assertions establish the specified identities, content and order; serialization requirements should also be checked before promoting an actual production artifact.

Capture commands, locale and preserved source bytes

Use a fresh working directory with GNU sort, uniq and comm, plus Python, installed separately. Save the following source as sort_lab.py, using UTF-8, LF line endings and a final newline. It installs nothing and contacts no production service. The Python 3.13 subprocess documentation explains argument sequences, stdout and stderr pipes, timeouts, explicit environments and process return codes; those are the driver’s responsibilities here.

from pathlib import Path
from datetime import datetime, timezone
import hashlib, json, os, platform, subprocess, sys, tempfile

ROWS = [
    b'B\tr3\tblue', b'A\tr2\tgreen', b'A\tr1\tred',
    b'A\tr2\tgreen', b'C\tr4\tgold',
]
EXACT = [b'A\tr1\tred', b'A\tr2\tgreen', b'B\tr3\tblue', b'C\tr4\tgold']
KEYED = [b'A\tr2\tgreen', b'B\tr3\tblue', b'C\tr4\tgold']
ENV = dict(os.environ, LC_ALL='C', LANG='C')

def require(ok, text):
    if not ok:
        raise AssertionError(text)

def encode(rows):
    return b''.join(row + b'\n' for row in rows)

def strings(rows):
    return [x.decode('ascii') for x in rows]

def main():
    folder = Path(tempfile.mkdtemp(
        prefix='sort-evidence-', dir=Path(__file__).parent
    ))
    report = {
        'status': 'FAIL',
        'started_utc': datetime.now(timezone.utc).isoformat(),
        'python': sys.version, 'platform': platform.platform(), 'locale': 'C',
        'source_sha256': hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),
        'input': strings(ROWS), 'independent_exact_manifest': strings(EXACT),
        'cases': [], 'commands': [],
    }

    def run(args, payload=None):
        p = subprocess.run(
            args, input=payload, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
            env=ENV, timeout=10, check=False
        )
        report['commands'].append({
            'argv': args,
            'stdin_sha256': (
                None if payload is None else hashlib.sha256(payload).hexdigest()
            ),
            'returncode': p.returncode,
            'stdout': p.stdout.decode('ascii'),
            'stderr': p.stderr.decode('ascii'),
        })
        require(p.returncode == 0, 'command failed: ' + repr(args))
        return p.stdout

    def record(name, actual, expected):
        row = {
            'id': name, 'actual': actual, 'expected': expected, 'status': 'FAIL'
        }
        report['cases'].append(row)
        require(actual == expected, name)
        row['status'] = 'PASS'

    try:
        report['versions'] = {
            tool: run([tool, '--version']).decode().splitlines()[0]
            for tool in ['sort', 'uniq', 'comm']
        }
        require('GNU coreutils' in report['versions']['sort'], 'GNU sort required')
        for row in ROWS:
            require(
                len(row.split(b'\t')) == 3 and b'\n' not in row, 'fixture schema'
            )
        key = ['sort', '-t', '\t', '-k1,1']
        bad = run(key + ['-u'], encode(ROWS)).splitlines()
        record('S1_key_unique', strings(bad), strings(KEYED))
        good = run(['sort', '-u'], encode(ROWS)).splitlines()
        record('S2_full_line_unique', strings(good), strings(EXACT))
        piped = run(['uniq'], run(key, encode(ROWS))).splitlines()
        record('S3_key_sort_then_uniq', strings(piped), strings(EXACT))
        stable = run(['uniq'], run(key + ['-s'], encode(ROWS))).splitlines()
        record(
            'S4_stable_sort_is_not_global_dedupe', strings(stable),
            strings([ROWS[1], ROWS[2], ROWS[3], ROWS[0], ROWS[4]])
        )
        changed = [ROWS[2], ROWS[0], ROWS[1], ROWS[3], ROWS[4]]
        other = run(key + ['-u'], encode(changed)).splitlines()
        record(
            'S5_input_order_changes_survivor', strings(other),
            strings([ROWS[2], ROWS[0], ROWS[4]])
        )
        expected_path = folder / 'expected.sorted'
        actual_path = folder / 'actual.sorted'
        expected_path.write_bytes(encode(EXACT))
        actual_path.write_bytes(run(['sort'], encode(bad)))
        missing = run([
            'comm', '--check-order', '-23', str(expected_path), str(actual_path)
        ]).splitlines()
        extra = run([
            'comm', '--check-order', '-13', str(expected_path), str(actual_path)
        ]).splitlines()
        record(
            'S6_reconcile_despite_zero_exit',
            {'missing': strings(missing), 'extra': strings(extra)},
            {'missing': ['A\tr1\tred'], 'extra': []}
        )
        groups = {}
        for row in good:
            groups.setdefault(row.split(b'\t')[0], []).append(row)
        conflicts = {k.decode(): strings(v) for k, v in groups.items() if len(v) > 1}
        record(
            'S7_key_conflict_quarantine', conflicts,
            {'A': ['A\tr1\tred', 'A\tr2\tgreen']}
        )
        approved = {b'A': b'r1', b'B': b'r3', b'C': b'r4'}
        selected = [
            row for row in good
            if row.split(b'\t')[1] == approved[row.split(b'\t')[0]]
        ]
        require(len(selected) == len(approved), 'approval incomplete')
        record(
            'S8_explicit_survivor_mapping', strings(selected),
            ['A\tr1\tred', 'B\tr3\tblue', 'C\tr4\tgold']
        )
        record('S9_declared_empty_input', run(['sort', '-u'], b'').decode(), '')
        clean = [ROWS[2], ROWS[0], ROWS[4]]
        record(
            'S10_unique_key_control', run(key + ['-u'], encode(clean)).decode(),
            encode(clean).decode()
        )
        require(len(report['cases']) == 10, 'coverage')
        report['status'] = 'PASS_10_CASES'
    except Exception as e:
        report['error'] = {'type': type(e).__name__, 'message': str(e)}
        raise
    finally:
        report['finished_utc'] = datetime.now(timezone.utc).isoformat()
        target = folder / 'results.json'
        target.write_text(json.dumps(report, indent=2) + '\n')
        print(target)

if name == '__main__':
    main()

Run from the directory containing the source:

python sort_lab.py

The SHA-256 of this source, saved with the line endings specified above, is d54bbf6ae0eebd521028698f1090147d311896e14151716b8361cf2f9c532e12.

The recorded run of this source produced PASS_10_CASES. Its start and finish timestamps identify an execution; they do not establish a throughput benchmark. The successful case count describes diagnostic coverage; it does not mean every candidate produced by the laboratory is approved for downstream use.

The report records runtime versions rather than assuming them. For each subprocess that completes in this ASCII laboratory, the driver records its argument vector, its standard-input SHA-256 when a byte payload is passed, the return code, decoded ASCII standard output and standard error. The environment applies LC_ALL=C and LANG=C to each subprocess. The Python documentation distinguishes process completion from output inspection: return code, stdout and stderr are separate properties of the completed process.

The evidence format has a specific boundary: comm receives file operands, so those command entries have no stdin payload and therefore record stdin_sha256 as null. Preserve expected.sorted and actual.sorted with the evidence directory, and hash those files separately in an operational package if the audit requirement is to bind every file operand cryptographically. Likewise, the report stores this ASCII fixture’s stdout content losslessly as decoded text rather than a second stdout digest. A source edit, including a formatting change, requires a new source hash and a new execution record.

A failed command is not a completed evidence package. When a subprocess returns a nonzero status, the driver records that result and then raises an exception. A launch failure or timeout can raise before a command entry is appended; the handler records the exception as a top-level error. In either case, the finally block attempts to save the partial report and print its path. Preserve available failure evidence instead of rerunning until only a green artifact remains.

Preserve multiplicity before reducing to a set

The original ledger records r2 twice and r1, r3 and r4 once each. The four-row oracle deliberately removes multiplicity only for exact-row acceptance. Keep both representations: the set-like manifest answers which distinct records remain, while the original ledger explains the five source occurrences.

This matters during retrospective recovery. A four-row exact-row artifact cannot tell you by itself whether r2 originally occurred twice, five times or once. The source ledger supplies that fact. If duplicate frequency has operational significance, preserve counts before reducing the publication population.

Retain source bytes, command evidence, manifests and approvals through the period when downstream corrections may be needed. A cryptographic hash identifies retained bytes; it cannot reconstruct a discarded source. If the original population becomes unavailable, the repaired command alone cannot establish what an earlier run lost.

Repair the pipeline without overwriting its only input

Repair begins with the selected contract and an intact source. For exact-row removal in this fixture, remove the first-field restriction from the uniqueness operation and validate the full-line candidate. If retaining the two-stage form, preserve the specific nonstable key sort and ordinary uniq combination tested in S3 rather than substituting other sort options.

For a one-row-per-key destination, the repair is broader than removing an option. Preserve distinct rows, identify collisions, and require either quarantine or an approved mapping before selection. A correctly implemented exact-row stage can still produce a dataset unsuitable for that destination until ownership is resolved.

Use a staged procedure:

1. Freeze the source and failed output. Preserve their batch identities, bytes and evidence before changing commands. Record which downstream artifact came from the defective stage.

2. Write a new candidate. Use a separate path and the chosen argument vector. Capture process status and actual output; do not truncate the only source through output redirection.

3. Reconcile against the appropriate expectation. Check missing records, extras, required occurrence counts and exact selected content. Include the collision or exclusion ledger when the policy requires it.

4. Review unresolved differences. A missing row without an approved exclusion blocks acceptance. Reaching the desired count does not make an artifact complete.

5. Promote the validated artifact. Change downstream references only after candidate, policy and evidence have been approved together. Retain the superseded artifact for investigation and correction.

Treat a command edit and an artifact replacement as separate changes. The command may be correct while its input belongs to the wrong batch, or a reviewed candidate may have been generated by a different command version. Bind the approved output to its input identity and recorded arguments so that the release decision refers to a specific result.

Replaying a retained source also requires the applicable historical contract. If policy changed after the original publication, decide whether a correction should reproduce the historical approved meaning or introduce a new dataset version. Do not silently present a new survivor decision as a technical restoration.

Historical recovery has a hard boundary. The defective three-row output does not contain A/r1/red; rerunning another sort against those three rows cannot reconstruct the absent content. Recovery requires the original source snapshot or another independently authoritative copy. If neither exists, HOLD the affected batch as unresolved rather than claiming that fixing the command repaired its historical data.

Finally, trace consumers of the incomplete artifact. A repaired staging file is insufficient if an export, report, cache or downstream batch still references the old version. Code repair prevents repetition; data recovery accounts for what was already published.

Select accept, repair, quarantine or hold

Assign a decision to the artifact under its named contract, not to the utility in isolation. The same three-row count can describe an unauthorized keyed result or an approved survivor population. Identity and approval determine the interpretation.

Quarantine describes the disposition of ambiguous data; HOLD describes the release boundary when the required decision cannot yet be made. They can coexist. A batch containing quarantined A records is not automatically approved as a B-and-C-only delivery.

A format or locale mismatch also prevents acceptance of this laboratory’s conclusion for the affected artifact. If the actual input violates the declared three-field normalized TSV format, HOLD that delivery and repair the input contract or processing stage. If the recorded locale differs, rerun the relevant checks in the declared environment before approving equivalence. Do not label an untested configuration defective solely because it differs from the laboratory.

S9 supplies an explicitly empty byte sequence and expects empty output. That utility behavior does not authorize an unexpectedly empty business batch. Require evidence that the intended source was received and complete before applying an empty-input policy. Missing input, failed acquisition and a declared empty delivery are different conditions.

The final ACCEPT test should be positive and explicit: both missing and extra complete-record differences are empty under the chosen manifest; required multiplicity checks pass; any business-key conflicts are resolved according to their approved disposition; and the candidate is bound to the intended source and command evidence. A green utility exit code is only one input to that decision.

Assign responsibility for past discarded rows

Name an owner for recovery separately from the maintainer implementing the command repair. Recovery includes locating retained inputs, establishing affected batches, reconstructing approved outputs and tracing consumers of incomplete artifacts. The business owner remains responsible for unresolved survivor decisions.

Close each affected batch with its own result: recovered and reconciled, awaiting approval, or unrecoverable from available evidence. Do not close the recovery task merely because future runs now use a corrected command. This preserves the distinction between preventing new losses and accounting for records already omitted.

Install regression tests for command changes

Retain all ten cases as a compact explanation of the failure boundary. Their expected identities and payloads make them more useful than a single test asserting that a repaired output has four lines. Each case protects a specific assumption or records a deliberate counterexample.

S10 deserves attention because it explains how a weak test suite could miss the problem. The keyed command appears harmless when every key already identifies one row. The absence of a collision in that control says nothing about the behavior demonstrated by the original A group.

S9 similarly checks an explicit boundary without simulating a missing file or interrupted upstream delivery. Do not expand its interpretation beyond what its supplied bytes establish. Add production-specific receipt and completeness checks when adopting the contract operationally rather than claiming this small laboratory already exercises them.

Run the suite when changing a key boundary, uniqueness flag, stability setting, stage composition or relevant runtime environment. Review argument vectors as behavior, not formatting. A one-character option change can invalidate the property on which the next stage depends.

Keep expectations independent of candidate output. When an assertion fails, retain the partial report and inspect actual versus expected content. Updating the expectation is appropriate only after an approved contract change or an explained correction to the test, with the original discrepancy preserved.

The driver also records versions instead of assuming that the executable found on PATH matches the preparation environment. A successful rerun on a different version would be additional bounded evidence. An unexplained divergence would require investigation; neither outcome justifies relabeling the binary as the version originally tested.

A passing diagnostic can deliberately confirm unsafe behavior. S1, S4, S5 and S6 are especially important because their PASS states mean “the expected failure mode was reproduced.” Do not collapse all ten cases into a production-readiness boolean. The release decision belongs to the chosen contract and its reconciled candidate.

These practices connect with reliable data-pipeline foundations, whose broader discussion emphasizes testing, validation and lineage. This regression suite stays focused on one local transformation. Its value is the explicit link between source population, command semantics and approved output. Ten passing cases establish ten checked observations, including unsafe controls; only reconciliation under the selected contract supports acceptance.

Strengthen pipeline validation with the Data Engineering Program

This exercise develops a practical engineering skill: translating a data promise into checks another person can inspect. The useful work is defining what a row means, tracing how a transformation treats it, and distinguishing a technical result from an authorized business decision.

That responsibility persists wherever data moves. The discussion of transformation responsibility in data pipelines provides broader context for why managed movement does not eliminate ownership of contracts, quality and governance. This laboratory keeps the evidence local: five source occurrences, four distinct records, an explicit collision and a separately approved mapping.

A useful practice review should let another engineer answer three questions from retained artifacts: what was allowed to disappear, what actually disappeared, and who approved any intentional selection. Being able to answer them is more meaningful than presenting an unexplained green test summary.

For structured study of the surrounding discipline, Refonte Learning’s Data Engineering Program lists a three-month format at 12–14 hours per week and names data pipelines, extract-transform-load (ETL) processes, batch and streaming ingestion, transformations, storage, security, Hadoop and Spark, plus governance and compliance controls. The page also describes practical work and a potential internship. The published curriculum does not confirm that this specific GNU sort, uniq and comm laboratory is taught.

Choose the contract before choosing the command. Preserve the source, retain ambiguous candidates, and approve a delivery only when its complete records reconcile with the expectation that the responsible owner actually authorized.