Keep PyArrow Schema Projection from Silently Dropping Source Columns

A three-row export can be structurally valid, type-valid, and still be incomplete relative to the DataFrame that entered the job.

Consider a source with three named fields: record_id, units, and source_note. Give pyarrow.Table.from_pandas() an explicit schema containing only record_id and units, and PyArrow 20.0.0 is documented to construct an output with that schema while ignoring additional DataFrame columns not named in it. That is legitimate projection behavior, not by itself an Arrow failure.

The operational question is therefore not merely, “Did conversion succeed?” It is: for every known source field, was it retained or explicitly approved for exclusion?

The supplied laboratory fixes that question to three scalar rows and two contracts. FULL retains all three fields. OLD retains record_id and units, leaving source_note in the source but outside the Arrow schema. The executed fixture, dated October 26, 2026, used CPython 3.12.14, pandas 2.2.3, PyArrow 20.0.0, and Linux 6.18.55; NumPy 2.3.5 was separately recorded in the unchanged prepared environment. It finished with exit status 0 and PASS_14_CASES.

That pass includes intentional rejection cases. It does not mean every successfully converted table is publishable. The policy outcomes are different: retain every declared field, accept a precisely approved projection, reject unexpected source drift before conversion, or hold an artifact when source completeness cannot be established.

The scope is deliberately narrow: scalar strings and integers, no index preservation, local in-memory Parquet, and no claim about nested values, metadata retention, schema evolution, distributed publication, or upstream fields already lost before this DataFrame existed.

Define the source-field retention contract

Start with an inventory that exists independently of the eventual Arrow table. If the output itself defines what the source “must have been,” a projection can erase the evidence needed to detect its own omission.

The fixed source oracle is:

This immediately separates two legitimate contracts. FULL says the published representation includes every declared source field. OLD says the representation is intentionally narrower. Neither policy should be inferred from conversion success.

A three-row input becoming a three-row output is useful evidence about row cardinality, but it says nothing about whether three source columns became two. Likewise, a table whose values are correct for record_id and units does not establish what happened to source_note.

This is also separate from choosing a DataFrame engine. Refonte Learning’s discussion of pandas and Polars migration tradeoffs addresses workload and engine decisions; this fixture assumes pandas is already the source representation and asks a narrower export-contract question.

For review purposes, write the retention rule before calling PyArrow:

Every unique source field must either appear in the declared output or match an independently approved exclusion manifest.

The phrase “independently approved” matters. Generating an exclusion list automatically from whatever extra columns happen to arrive would convert drift detection into drift acceptance.

Read the explicit-schema projection rule

The central behavior is directly documented, not inferred from a surprising test result.

In the PyArrow 20.0.0 Table.from_pandas documentation, passing a schema means the output has that schema. Fields required by that schema but unavailable from the DataFrame or its index produce an error, while additional DataFrame columns or index levels not specified by the schema are ignored. The same reference describes safe=True as checking overflows and other unsafe conversions.

Those statements describe two different dimensions:

schema=OLD is a selection plus typing instruction. It says which output fields to construct and their Arrow types.

safe=True is a conversion-safety control. It is not documented as “reject any DataFrame column not named in the schema.”

The distinction explains why a conversion can safely produce the wrong artifact for a stricter business retention policy. There may be no invalid cast, no malformed array, and no missing field required by OLD. source_note simply lies outside that output contract.

Table.validate(full=True) is another output-side control. PyArrow documents validate as performing validation checks on the table, with full=True enabling more thorough checks. It raises if those table checks fail. Nothing in that API definition makes the original pandas field inventory or an organization’s retention decision part of validation.

Separate field accounting from type validation

A reliable export gate should keep at least four pieces of evidence separate.

A stricter cast cannot answer the first two questions. It can reject a bad units value while still permitting an unrelated source_note to be projected away.

That is the core operational lesson behind PyArrow schema extra columns: validation of what was produced and accounting for what was supplied are complementary controls, not substitutes.

Lock the local fixture and dependency versions

The supplied reference execution is version-specific evidence. It is not a claim about the newest PyArrow release or about every future exception type.

The recorded run used CPython 3.12.14, pandas 2.2.3, PyArrow 20.0.0, and Linux kernel 6.18.55. NumPy 2.3.5 was inspected separately in the unchanged prepared environment. Arrow’s pandas integration documentation states that preserve_index=False prevents storage of the pandas index; the fixture uses that setting intentionally, keeping the index outside this contract.

A reader can construct an isolated reproduction environment with:

python3.12 -m venv .venv
.venv/bin/python -m pip install numpy==2.3.5 pandas==2.2.3 pyarrow==20.0.0
.venv/bin/python -u arrow_schema_fixture.py > arrow_schema_results.json

That command is a reproduction procedure. The supplied evidence says the reference run used an already available pandas/NumPy environment plus an isolated PyArrow installation; it does not say this exact virtual-environment procedure generated the reference result. After redirecting output, inspect the process exit status and parsed JSON rather than treating the mere presence of arrow_schema_results.json as success.

This pin also avoids importing conclusions from later pandas behavior. Refonte Learning separately discusses pandas version changes that affect existing code; that migration context should not be retroactively applied to this pandas 2.2.3 fixture.

Save the supplied complete source as arrow_schema_fixture.py:

import datetime
import hashlib
import json
import platform
import sys
from pathlib import Path

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

ROWS = [
    {"record_id": "R1", "units": 4, "source_note": "alpha"},
    {"record_id": "R2", "units": 0, "source_note": "beta"},
    {"record_id": "R3", "units": 6, "source_note": "gamma"},
]
OLD = pa.schema([("record_id", pa.string()), ("units", pa.int64())])
FULL = pa.schema([("record_id", pa.string()), ("units", pa.int64()), ("source_note", pa.string())])
DROP = {"source_note": "synthetic-policy-v1: output intentionally omits the source note"}
report = {
    "status": "FAIL", "started_utc": datetime.datetime.now(datetime.timezone.utc).isoformat(),
    "python": platform.python_version(), "pandas": pd.__version__, "pyarrow": pa.__version__,
    "platform": platform.platform(), "source_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),
    "cases": [],
}
conversions = 0


class ContractError(ValueError):
    pass


def frame():
    return pd.DataFrame(ROWS, columns=["record_id", "units", "source_note"])


def convert(df, schema=None, safe=True):
    return pa.Table.from_pandas(df, schema=schema, preserve_index=False, safe=safe)


def summary(table):
    return {"columns": table.column_names, "rows": table.num_rows, "values": table.to_pylist()}


def add(case, observed, decision):
    report["cases"].append({"id": case, "observed": observed, "decision": decision})


def rejected(case, call, expected_type):
    try:
        call()
    except expected_type as error:
        add(case, {"error_type": type(error).__name__, "message": str(error)}, "REJECT_CANDIDATE")
    else:
        raise AssertionError(case + ": expected an exception")


def qualify(df, schema, approved_drops=None):
    global conversions
    approved_drops = {} if approved_drops is None else dict(approved_drops)
    if not all(isinstance(name, str) for name in df.columns) or not df.columns.is_unique:
        raise ContractError("Source column names must be unique strings")
    names = list(schema.names)
    if len(names) != len(set(names)):
        raise ContractError("Output schema field names must be unique")
    missing = sorted(set(names) - set(df.columns))
    extras = sorted(set(df.columns) - set(names))
    if missing:
        raise ContractError("Missing source fields: " + repr(missing))
    if set(extras) != set(approved_drops) or any(not isinstance(reason, str) or not reason.strip() for reason in approved_drops.values()):
        raise ContractError("Unapproved or stale exclusion manifest: " + repr(extras))
    before = df.copy(deep=True)  # Scalar values only in this fixture.
    projected = df.loc[:, names]
    expected_values = projected.to_dict(orient="records")
    conversions += 1
    table = convert(projected, schema=schema)
    table.validate(full=True)
    assert table.schema.remove_metadata().equals(schema.remove_metadata())
    assert table.num_rows == len(df)
    assert table.to_pylist() == expected_values
    pd.testing.assert_frame_equal(df, before)
    return table, {"source_columns": list(df.columns), "output_columns": names, "approved_exclusions": approved_drops}


def parquet_roundtrip(table):
    sink = pa.BufferOutputStream()
    pq.write_table(table, sink)
    return pq.read_table(pa.BufferReader(sink.getvalue()))


try:
    source = frame()
    baseline = source.copy(deep=True)
    complete = convert(source)
    assert complete.column_names == list(FULL.names) and complete.to_pylist() == ROWS
    add("A1", summary(complete), "ACCEPT_INFERRED_CONTROL")

    projected = convert(source, OLD)
    projected.validate(full=True)
    assert projected.schema.remove_metadata().equals(OLD)
    assert projected.column_names == list(OLD.names) and projected.num_rows == 3
    add("A2", summary(projected), "HOLD_UNAPPROVED_COLUMN_LOSS")

    unsafe = convert(source, OLD, safe=False)
    assert unsafe.equals(projected)
    add("A3", summary(unsafe), "REPAIR_SAFE_FLAG_ASSUMPTION")

    reordered = convert(source[["source_note", "units", "record_id"]], OLD)
    assert reordered.to_pylist() == projected.to_pylist()
    add("A4", summary(reordered), "HOLD_UNAPPROVED_COLUMN_LOSS")

    rejected("A5", lambda: convert(source.drop(columns="units"), OLD), KeyError)
    invalid = source.copy(deep=True); invalid["units"] = [4, "bad", 6]
    rejected("A6", lambda: convert(invalid, OLD), (pa.ArrowInvalid, pa.ArrowTypeError))

    duplicate = pd.DataFrame([["R1", 4, 5]], columns=["record_id", "units", "units"])
    before_calls = conversions
    rejected("A7", lambda: qualify(duplicate, OLD), ContractError)
    assert conversions == before_calls

    rejected("A8", lambda: qualify(source, OLD), ContractError)
    assert conversions == before_calls

    retained, ledger = qualify(source, FULL)
    assert retained.to_pylist() == ROWS
    add("A9", {"table": summary(retained), "ledger": ledger}, "ACCEPT_ALL_DECLARED_FIELDS")

    approved, ledger = qualify(source, OLD, DROP)
    assert approved.to_pylist() == projected.to_pylist()
    add("A10", {"table": summary(approved), "ledger": ledger}, "ACCEPT_APPROVED_PROJECTION")

    expanded = source.assign(unexpected=["x", "y", "z"])
    before_calls = conversions
    rejected("A11", lambda: qualify(expanded, OLD, DROP), ContractError)
    rejected("A12", lambda: qualify(source.drop(columns="record_id"), FULL), ContractError)
    assert conversions == before_calls

    saved_projection = parquet_roundtrip(projected)
    assert saved_projection.column_names == list(OLD.names)
    assert saved_projection.to_pylist() == projected.to_pylist()
    add("A13", summary(saved_projection), "HOLD_SOURCE_COMPLETENESS_UNPROVEN")

    saved_complete = parquet_roundtrip(retained)
    assert saved_complete.column_names == list(FULL.names) and saved_complete.to_pylist() == ROWS
    add("A14", summary(saved_complete), "ACCEPT_DECLARED_ROUNDTRIP")

    pd.testing.assert_frame_equal(source, baseline)
    assert [row["id"] for row in report["cases"]] == [f"A{i}" for i in range(1, 15)]
    report["successful_qualified_conversions"] = conversions
    report["status"] = "PASS_14_CASES"
except Exception as error:
    report["failure"] = {"type": type(error).__name__, "message": str(error)}
    raise
finally:
    report["finished_utc"] = datetime.datetime.now(datetime.timezone.utc).isoformat()
    print(json.dumps(report, indent=2))

The supplied SHA-256 for that source is 8f22339eceb6e7f72b11c89184914b477847bc3b3b50e803658ea3633d0e02a3. That hash identifies the source used for the reported evidence; it does not itself prove the semantic correctness of any export.

Reproduce a valid output with a missing source column

A1 establishes the positive control without an explicit schema. The source DataFrame is constructed from ROWS with fields in this order:

record_id, units, source_note

The inferred Arrow table has those same three columns, three rows, and the exact fixed values. That tells us the fixture can carry the complete source through the basic pandas-to-Arrow path.

A2 changes one thing: it calls conversion with OLD:

OLD = pa.schema([
    ("record_id", pa.string()),
    ("units", pa.int64()),
])

The original source still contains all three fields. The resulting table contains:

source_note is absent.

The result then passes projected.validate(full=True), has three rows, and its metadata-stripped schema equals metadata-stripped OLD. That observed result is exactly compatible with the documented explicit-schema rule: additional DataFrame fields not represented in the supplied schema are ignored.

The schema equality needs careful wording. PyArrow’s Schema.equals reference states that schema equality can optionally include key/value metadata through check_metadata, which defaults to false. This fixture goes further and explicitly calls remove_metadata() on both sides before comparing. Consequently, A2 establishes equality of the tested field structure and types, not metadata preservation.

The unchanged row count is equally narrow evidence. It proves that three projected records reached the Arrow table. It does not prove that every source field did.

Most importantly, source_note is already gone before the Parquet writer appears anywhere in the path. A later file-format roundtrip cannot be blamed for this omission.

The independent oracle is ROWS, not projected.to_pylist(). Once a field has been removed, asking the projected table what the original source contained is circular. A2 is therefore marked HOLD_UNAPPROVED_COLUMN_LOSS: the table is valid under OLD, but the job has not yet supplied policy evidence authorizing the exclusion of a known source field.

Contrast extra fields with missing and incompatible fields

A2 must not be confused with A5 or A6. They exercise different failure categories.

A3 repeats OLD with safe=False. The observed output equals A2. That is useful because it repairs a common misconception: safe=True did not prevent source_note from being omitted, and turning safety off does not somehow expose or retain it. The Arrow documentation defines the flag around unsafe conversions such as overflow, not extra-field rejection.

A4 supplies the same three source fields in the order:

source_note, units, record_id

and still obtains the same two-column OLD output values. Under this fixture, source ordering is therefore not the retention control. Field identity is name-based, and the declared output names determine what survives. The policy later reinforces that behavior with df.loc[:, names].

A5 removes units, which OLD requires. The supplied run observed KeyError. That is a materially different condition: the requested output field is unavailable. Arrow 20.0.0 documentation says schema fields not found in the DataFrame columns or index raise an error, but this does not establish that every future PyArrow version will express that situation with the same Python exception text or type.

A6 puts "bad" into the otherwise integer units field. The reference execution observed ArrowInvalid. Again, that is evidence for this pinned environment, not a universal error-message contract.

Why a conversion error in one case is insufficient

It is tempting to demonstrate A5 or A6 and conclude that “the schema protects the export.” That conclusion overgeneralizes one branch.

A missing schema-required field tests whether PyArrow can construct the requested output.

An incompatible scalar tests whether selected values can satisfy the requested Arrow type.

An extra source field tests neither. Under the documented explicit-schema behavior, it can simply be outside the projection.

A robust gate therefore needs positive and negative controls for each risk it claims to detect. A type error in units does not demonstrate that source_note would be retained. Likewise, changing safe is not an appropriate remediation for completeness. Preserve safe=True when its conversion checks are desired; add independent source-field reconciliation for the retention question.

Reject ambiguous labels before reconciliation

Set arithmetic only becomes trustworthy after the names being compared have unambiguous identities.

pandas explicitly permits duplicate row or column labels. Its pandas 2.2.3 duplicate-label documentation notes that labels are not required to be unique and shows that duplicate labels can alter or invalidate ordinary operations.

That matters because this expression:

set(df.columns)

discards multiplicity. A DataFrame with:

record_id, units, units

becomes the set:

{"record_id", "units"}

A naive reconciliation could therefore report that its source names exactly match OLD, even though there are two different columns carrying the same label. The problem is not that sets are wrong; it is that their preconditions were not established.

A7 addresses this before any qualified Arrow conversion. Its DataFrame contains one row and duplicate units labels. qualify() requires every source label to be a string and df.columns.is_unique to be true. The duplicate candidate raises the synthetic local ContractError, and the fixture verifies that the qualified-conversion counter did not increase.

The same principle applies to the Arrow schema side. qualify() converts schema.names to a list and rejects the schema if its name count differs from the size of the corresponding set. That is a policy imposed by this laboratory so that each field identity has one unambiguous name.

A related Refonte Learning article on feature identity and scikit-learn column checks already warns that a ColumnTransformer configured with remainder="drop" can accept a wider DataFrame while unspecified columns are dropped. That is a useful neighboring lesson, but this export problem is different: no prediction coordinates or fitted model are involved. The question here is whether every known source field is either exported or explicitly accounted for.

Check names before computing missing and extra sets

The fixture’s ordering is intentional:

if not all(isinstance(name, str) for name in df.columns) or not df.columns.is_unique:
    raise ContractError(...)

names = list(schema.names)
if len(names) != len(set(names)):
    raise ContractError(...)

missing = sorted(set(names) - set(df.columns))
extras = sorted(set(df.columns) - set(names))

Only after uniqueness and string-label requirements hold do missing and extras represent meaningful field identities.

That guard is deliberately narrower than a universal DataFrame validator. It says nothing about nested objects, categoricals, time zones, MultiIndex semantics, semantic units, upstream lineage, or whether "units" has the correct business definition. It simply establishes enough naming discipline for this bounded source/output reconciliation.

Reconcile the source with the intended output

qualify() is the policy boundary that the raw convert() function lacks.

For an output schema, it derives names = list(schema.names). It then computes:

missing = sorted(set(names) - set(df.columns))
extras = sorted(set(df.columns) - set(names))

Those two lists have different meanings.

missing means the export contract requires a field that the source cannot provide.

extras means the source contains a known field that this output contract does not retain.

The strict case is A8:

qualify(source, OLD)

source contains record_id, units, and source_note; OLD contains the first two; no exclusion manifest is supplied. Therefore:

missing = []
extras = ["source_note"]
approved_drops = {}

The candidate raises ContractError because the exact extra-field set is not identical to the exact approved-drop set. The fixture also checks that the conversion counter has not moved. This is a useful release-gate property: an unqualified candidate should not first be projected and then inspected after the evidence has already disappeared.

The strict rule does not mean every input field must always be published. It means omission must be a deliberate state transition rather than an incidental side effect of the conversion API.

After reconciliation succeeds, qualify() performs an explicit selection:

projected = df.loc[:, names]

It then derives expected scalar records from that approved projection, calls Table.from_pandas, validates the resulting table, checks field structure and row count, compares exact values, and verifies that the source DataFrame still equals its pre-conversion copy.

Make every permitted exclusion explicit

The lab’s approved exclusion is:

DROP = {
    "source_note":
        "synthetic-policy-v1: output intentionally omits the source note"
}

Operationally, an exclusion record should identify at least the field, the reason, and a policy revision or equivalent decision context. The string here contains those ideas compactly.

But the fixture is not an approval system. Any nonblank string satisfies its local check. No authentication, authorization, legal retention review, ticket signature, or owner identity is verified. A production process would need a separate mechanism establishing who has authority to approve an exclusion and which policy revision they approved.

Most importantly, never construct the approval manifest from extras after discovering them. Code such as “for every extra column, create an exclusion reason automatically” makes the check tautological. The expected exclusions must predate the candidate being evaluated, or be separately reviewed as a contract change.

Preserve all declared fields in the accepted candidate

A9 is the clean all-fields outcome:

retained, ledger = qualify(source, FULL)

FULL declares:

record_id: string
units: int64
source_note: string

The source has exactly those three names. missing is empty, extras is empty, and therefore the empty default exclusion manifest is exactly correct.

The accepted ledger records:

source_columns:
  record_id, units, source_note

output_columns:
  record_id, units, source_note

approved_exclusions:
  {}

After name reconciliation, the selected DataFrame is converted. table.validate(full=True) succeeds; metadata-stripped schema equality succeeds; the row count is three; and table.to_pylist() equals the independently defined three projected records. Because the projection in this case includes all source fields, those expected records are the original ROWS values.

The fixture also snapshots the scalar DataFrame before the operation and uses pd.testing.assert_frame_equal() afterward. At the end of all cases it compares source against the earlier baseline again. For this bounded fixture, that is useful evidence that the source frame was not altered while generating candidates.

Do not extend that claim beyond the data being tested. DataFrame.copy(deep=True) is not being offered here as a universal recursive isolation mechanism for arbitrary nested Python objects. The three fixture fields contain scalar strings and integers only.

Two other dimensions are intentionally outside the acceptance result.

First, preserve_index=False means the pandas index is excluded by design; Arrow’s integration documentation explicitly provides that option for not storing index data.

Second, schema metadata is removed before equality. As the Schema.equals documentation makes clear, metadata comparison is a distinct option, so A9 must not be advertised as metadata-retention evidence.

The result remains valuable precisely because its claim is narrow: for the declared scalar fields entering this job, the accepted candidate preserves the expected names, row identities, and values.

Approve projection as a separate policy outcome

A10 demonstrates why not every two-column output should be called data loss.

It runs:

qualify(source, OLD, DROP)

The source still contains source_note, so extras == ["source_note"]. This time the approved manifest contains exactly that field with a nonblank synthetic reason. The field sets agree, so the job is permitted to construct the two-column candidate.

The resulting Arrow values are the same as A2:

R1, 4
R2, 0
R3, 6

Yet A2 and A10 have different release meanings. A2 reached the same representation through direct explicit-schema projection without field-accounting evidence and is marked HOLD_UNAPPROVED_COLUMN_LOSS. A10 reached it after reconciling the complete source and exact exclusion set, so the fixture records ACCEPT_APPROVED_PROJECTION.

That distinction is the heart of an approved column exclusion manifest. The bytes or Arrow values alone cannot tell a reviewer whether an absent field was intentionally excluded.

A11 then expands the source with:

unexpected = x, y, z

while leaving DROP unchanged. The actual extras are now source_note and unexpected, whereas the manifest authorizes only source_note. qualify() rejects the candidate before conversion. Existing approval does not silently cover future drift.

A12 tests the opposite direction: it removes record_id and asks for FULL. missing becomes ["record_id"], so the candidate is rejected before conversion.

The policy therefore supports intentional narrowing without turning projection into a wildcard permission. Reordering an unchanged field set is tolerated by the name-based selection; changing the set requires a new explicit decision.

Compare Parquet readback with the original source ledger

Parquet adds a persistence boundary, but it cannot reconstruct information that never reached the table passed to the writer.

PyArrow 20.0.0 documents pyarrow.parquet.write_table as writing a supplied Arrow Table to Parquet. The input to that operation is the table itself.

Likewise, pyarrow.parquet.read_table reads Parquet into an Arrow table and explicitly supports pyarrow.BufferReader for bytes or buffer-like data. That is the local in-memory path used by this fixture.

A13 takes projected, which came from the unapproved raw A2 projection, writes it to a BufferOutputStream, and reads it back:

saved_projection = parquet_roundtrip(projected)

The roundtrip succeeds. The returned table has three rows, exactly two columns, and the same record_id and units values as the input table.

That is good Parquet consistency evidence. It is not source-completeness evidence.

source_note was absent from projected before pq.write_table() was called. A successful writer cannot infer those missing values from the original pandas DataFrame because that DataFrame is not its input. A successful reader similarly cannot rediscover a field that was never in the serialized table.

A14 performs the same write/read operation on retained, the FULL candidate qualified in A9. The readback contains all three fields and equals the fixed ROWS oracle. That supports ACCEPT_DECLARED_ROUNDTRIP for the bounded contract.

Refonte Learning’s discussion of field preservation across a Protobuf relay examines a different technology and mixed-version boundary, but it illustrates a related evidence discipline: successful parsing or transport is weaker than proving survival of the field whose preservation matters. This Arrow article applies that discipline to a local pandas-to-Arrow projection instead of a protocol relay.

Keep output consistency separate from source completeness

The evidence can be partitioned cleanly.

Artifact-to-table checks ask whether Parquet readback corresponds to the Arrow table that was written.

Table-to-contract checks ask whether the Arrow table contains the field set authorized by the export contract.

Contract-to-source checks ask whether every independently known source field is either represented by that contract or explicitly approved for exclusion.

A13 passes the first comparison and lacks sufficient evidence for the third. A14 passes the roundtrip comparison after A9 has already established the relevant source-field contract.

That is why “Parquet roundtrip passed” should never be used as shorthand for “nothing from the source was lost.”

Capture the full acceptance and rejection matrix

The complete laboratory matters because no single case proves the whole policy.

The supplied execution recorded source SHA-256 8f22339eceb6e7f72b11c89184914b477847bc3b3b50e803658ea3633d0e02a3, CPython 3.12.14, pandas 2.2.3, PyArrow 20.0.0, Linux 6.18.55, and two successful calls through the qualified conversion path. It completed with PASS_14_CASES.

The matrix is:

Several review traps disappear once the cases are displayed together.

A2 and A13 show that output validity and serialization stability do not establish completeness.

A5 and A6 show that failures are possible for missing or incompatible required data, but those failures do not imply rejection of extras.

A8, A11, and A12 show that the policy can stop a candidate before the qualified conversion is invoked.

A9 and A10 establish two different valid business outcomes: preserve all declared source fields, or intentionally project a specific field under an exact exclusion decision.

Finally, PASS_14_CASES means the experiment behaved as designed, including expected negative cases. It is not an instruction to approve A2, A5, A6, A7, A8, A11, A12, or A13 for publication. Test-suite success and candidate-release approval are different status dimensions.

Recover an export whose fields were already omitted

Once an artifact has been produced without a source field, recovery should begin from evidence independent of that artifact. The executed lab has no real publication step, so the following is an operational procedure rather than an observed incident response.

1. Isolate the candidate. Do not promote a projected Arrow or Parquet artifact merely because its schema, row count, or readback is internally consistent. Record its field names, hash or artifact identity where appropriate, generating job revision, and conversion status.

2. Locate an independent source. Prefer the DataFrame input retained for the run, a reproducible upstream snapshot, or another authoritative source capable of establishing the original names and values. The already projected table is not an oracle for omitted fields.

3. Reconstruct the intended field ledger. List the independently known source names, output-contract names, and any exclusions that were actually authorized. Do not infer authorization from what happens to be absent from the artifact.

4. Rebuild the candidate. Where the required source values remain available, execute reconciliation again, then perform the qualified conversion and the appropriate readback test.

5. Record the decision. Accept a complete candidate, accept an independently approved projection, reject demonstrated drift, or hold the export when the original completeness question cannot be answered.

This procedure intentionally avoids promising “automatic rollback.” Data representation and information availability are separate concerns.

Explain the limits of rollback and reconstruction

Changing OLD back to FULL does not recreate source_note values if the only surviving object is an OLD table or Parquet representation. A schema describes fields and types; it is not a backup of values that were never written.

Recovery therefore has three materially different outcomes.

Where retained source data still contains alpha, beta, and gamma, rebuild the complete export and rerun the checks.

Where the exclusion was genuinely authorized and evidence of that authorization exists, retain the narrower artifact as an approved projection.

Where the original source inventory or omitted values cannot be established independently, use HOLD. Do not convert absence of evidence into a claim of losslessness, and do not fabricate missing scalar values from conventions or downstream expectations.

The ledger in this fixture starts at the DataFrame supplied to the job. Even a perfect A9-style result cannot prove that some earlier ingestion layer did not discard a field before this DataFrame was created. Upstream preservation requires a separately defined boundary and oracle.

Assign ownership and a release decision

A useful completeness gate has named responsibilities because field intent, schema mechanics, and job execution are different kinds of authority.

The dataset owner should define what fields arrived at this export boundary and what they mean. That owner is the appropriate source for deciding whether source_note is required in the publication or eligible for exclusion; the conversion API cannot make that business decision.

The schema maintainer owns the declared Arrow output shape and types. Their review should distinguish “this two-field schema is intentional” from “this is an old schema that happened not to know about a third field.” Those states can produce identical Arrow tables while requiring opposite release decisions.

The export-job owner owns enforcement. The job should reject non-string or duplicate labels before set reconciliation, calculate missing and extra names, compare extras to the predeclared manifest, select fields by approved names, convert safely, validate the resulting table, compare deterministic values, and retain enough evidence for review.

For this fixture, record identities provide a simple value oracle. Reviewers can see that R1, R2, and R3 still map to units 4, 0, and 6, and, under FULL, to notes alpha, beta, and gamma. In a larger implementation, deterministic checking may use a different representation, but it should still be traceable to an independent source contract rather than an already-projected artifact.

A practical release record should contain the explicit retained names, explicit excluded names, policy revision, candidate conversion result, readback result where Parquet is in scope, fixture or source identity, dependency versions, and reviewer decision. A source hash proves which fixture code generated evidence; it does not substitute for field-value assertions.

Keep the boundary honest. This three-row laboratory does not qualify transactional publication, concurrent producers, distributed commits, richer nested types, categorical or timezone semantics, metadata retention, indexes, or large-data workload behavior. Those need their own acceptance work instead of being smuggled under the phrase “the schema test passed.”

The final decision rule is deliberately conservative: ACCEPT when retention or an exact approved projection is established; REJECT when known contract evidence demonstrates an invalid candidate; HOLD when completeness cannot yet be established. HOLD is not indecision. It is the correct status when the evidence cannot support either approval or a definitive reconstruction.

Practice field-level evidence in data engineering

The useful skill here is not memorizing that one API can ignore additional columns. It is learning to separate source inventory, intended projection, type validation, artifact validation, and release authority.

That discipline transfers directly to ordinary ETL and pipeline work. Refonte Learning’s broader guide to data engineering tools and their roles places ETL, integration, processing, storage, and pipeline tooling in the wider data-engineering landscape. This fixture narrows that broad practice to one reviewable conversion boundary.

Refonte Learning’s Data Engineering program is listed as a three-month program with a 12–14 hour weekly dedication. Its published educational path includes Introduction to Data Engineering, Data Warehousing and ETL Processes, and Big Data Technologies and Data Pipeline Design, with Hadoop and Spark named in the pipeline curriculum; the page also identifies ingestion, transformation, storage-related competencies, governance, and data protection among its areas of study.

Those are appropriate foundations for the habit this exercise develops: before accepting an exported representation, account for each field at the boundary and preserve evidence of why it is present or why it is intentionally absent.