The playbook below demonstrates how to audit pandas’ DataFrame.explode for special cases like empty lists, scalars, nulls, duplicate indices, and to trace row provenance. We define the admission contract (what outputs should be allowed), then pin a reproducible environment and run synthetic tests. The key question is: if explode returns a row for an empty list, is that correct? We do not simply trust a green check; instead we compare actual output against an independent expected-case manifest and classify the result as ACCEPT / REPAIR / RERUN / HOLD. For each test case, record expected behavior from the documentation alongside the actual observed behavior and supporting evidence. The examples in this playbook are EXPECTED / NOT EXECUTED and require a local run before a release decision. Resolve any discrepancy before deployment.
1. Define the exact admission contract
Before running any code, specify exactly what pandas explode should do in each case. In data cleaning, we first “preserve the raw data and write a data contract first”. Here the contract must state, for example: if a column value is a nonempty list of length N, then explode should produce N rows; if the list is empty, the output should include a row with NaN (per pandas docs); if the value is a scalar or missing (None/NaN), it should remain as one row unchanged; duplicate index values in the input should reappear (replicated) for each element of an exploded list. Concretely:
Non-list scalar: e.g. an integer or string. Explode should treat it as a “single-element list” and leave it unchanged in one row.
List with elements: e.g. [x,y,z]. Explode should yield three rows (one per element), each carrying the original index and copying other columns.
Empty list ([]): Pandas documents that empty list-like values produce one row with NaN in that column. (The title question “Was There an Element to Expand?” highlights that pandas does not drop the row by default.)
Missing value (None or NaN): This is not a list, so it should be treated as a scalar. We expect one row with a missing value (often displayed as NaN) in that column.
Multiple columns explode: If exploding more than one column together, each specified column must have lists of equal length in each row, or a ValueError is raised. If lengths match, each list element pairs with the element in the other column. An empty list does not broadcast to match a longer list: pairing [] with ['x', 'y'] raises ValueError because their expansion counts differ.
Duplicate index: The original index values should carry over. For example, if the input index is [0,1,1,2], and row 1 has a list of length 2, then the output will contain two rows both labeled index 1 (matching the original row). This preserves the original index labels. If labels are duplicated, retain a unique source-row identifier as well; the index alone cannot distinguish those input rows.
These rules form the oracle: the ground-truth output for each input scenario. We will compare pandas’ actual behavior against this contract. All decisions (ACCEPT vs REPAIR etc.) depend on whether the observed output matches this admitted contract.
2. Capture baseline versions and state
Pin the software versions and schema before testing. Use a reproducible Python environment (e.g. a virtualenv or Docker container) with known versions of pandas, Python, and any dependencies. This avoids unexpected changes when versions update. For example, in a published pandas GroupBy audit we fixed “CPython 3.13.5 with pandas 2.2.3” as the reference environment so any reader can reproduce results exactly. In our case, we might choose (for example) Python 3.13 and pandas 3.0.x. Replace these illustrative version ranges with exact patch versions in the evidence record. Using pip freeze (or similar) to list all package versions and including them in logs is recommended. Also capture the operating system or platform if relevant.
For example, one could run:
import sys, pandas as pd
print("Python:", sys.version)
print("pandas:", pd.__version__)Illustrative version output (EXPECTED / NOT EXECUTED):
Python: 3.13.x ...
pandas: 3.0.xPin environment: freeze these versions in the fixture definition. If the tests are rerun on a different pandas version, treat it as a new test condition and compare results again. (Pandas 3.x became available in 2026, and requires Python 3.11+, so ensure your environment meets those requirements.)
Pin environment and versions
Test case: Run pip freeze or conda list to capture the exact versions of pandas, Python, NumPy, etc. For instance, record the exact pandas 3.0.x patch version selected.
Oracle: The recorded versions match the chosen baseline (e.g. CPython 3.13.5 and pandas 2.2.3 in GroupBy example).
Rejection: If versions differ (or ignore_index behavior changed), we must RERUN tests in a matching environment or document the version difference.
3. Build a disposable isolated fixture
Create a small, synthetic DataFrame for testing. Do not use real data; use minimal examples that cover each case. For example, construct a DataFrame with one or two columns: one column containing a variety of list-like and scalar values, and perhaps another static column. Use fixed values (no randomness) so the test is repeatable.
We draw an analogy to the Shapely geometry playbook, which used a “fixed synthetic planar fixture” of representative shapes. Likewise, we define our pandas fixture in code. For instance:
import pandas as pd
df_fixture = pd.DataFrame({
'id': [0, 1, 2, 3],
'list_col': [[10, 20], [], 'foo', None],
'value': [1, 2, 3, 4] # some other column to copy
})
print(df_fixture)Expected input:
id list_col value
0 0 [10, 20] 1
1 1 [] 2
2 2 foo 3
3 3 None 4(This synthetic table covers all the special cases: a list of two elements, an empty list, a scalar string, and a None.) Because the fixture is disposable, we keep it in memory and recreate it fresh for each test to avoid side effects. After testing we discard it (no persistent writes).
Preserve a negative control
Test case: Include a DataFrame where the explode column has only scalars (no lists). For example: df_control = pd.DataFrame({'x': ['a', 'b', 'c']}). Exploding it (df_control.explode('x')) should leave the DataFrame unchanged.
Oracle: Output equals input (no change in row count or values).
Rejection: If explode altered or dropped rows, that violates the contract. This negative control checks that non-list values are handled as scalars with no unintended side effects.
4. Create an independent expected-case manifest
List each test scenario with its expected output (based only on the contract, not on code). For clarity, present each case’s input and expected output in table or list form. All cases below are EXPECTED / NOT EXECUTED:
Case A: Single-element list. Input: {'id': 0, 'list_col': [100]}. Expected output: one row {'id':0, 'list_col':100} (same as the element), with index 0.
Case B: Multi-element list. Input: {'id': 0, 'list_col': [1,2,3]}. Expected: three rows, each with id=0 and list_col=1, then 2, then 3 (index 0 repeated thrice).
Case C: Empty list. Input: {'id': 1, 'list_col': []}. Contract says output should still have one row with id=1 and list_col=NaN.
Case D: Scalar value. Input: {'id': 2, 'list_col': 'foo'}. Expected: one row unchanged (id=2, list_col='foo').
Case E: Null value. Input: {'id': 3, 'list_col': None}. Expected: one row {'id':3,'list_col':None}; the scalar remains None and is recognized as missing.
Case F: Duplicate index with list. Input: two rows both index=0: [ [5], [6] ]. Exploding each: we should get two rows (index=0 twice) with values 5 and 6.
Case G: Multi-column explode. Input: {'A':[ [1,2], [3] ], 'B':[ ['a','b'], ['c'] ]}. Both lists are length-matched. Expected: row1 splits into (1, 'a') and (2, 'b') (index 0 duplicated), row2 yields (3, 'c') (index 1). If one list were empty (e.g. A=[]) while B=['x','y'], the unequal expansion counts would raise ValueError; pandas does not broadcast NaN across the two elements.
This manifest is independent of running code; it comes directly from our understanding of the contract (and pandas docs). We will use it to assert correctness.
Preserve a negative control
We ensure at least one test with “no lists” is included above as a control. The manifest should note that exploding purely scalar rows yields the original row count.
5. Reproduce the misleading success
Now execute the code and record the raw output. Import pandas in the pinned environment and call explode. We expect no exceptions, but must inspect the results. For example:
result = df_fixture.explode('list_col')
print(result) # NOT EXECUTED (EXPECTED output shown below)Expected output for our fixture (NOT EXECUTED):
id list_col value
0 0 10 1
0 0 20 1
1 1 NaN 2
2 2 foo 3
3 3 None 4According to pandas docs, this matches the rules: the list [10, 20] is expected to split into two rows (id=0 repeated), the empty list is expected to produce a row with NaN in list_col, and the scalar string and None each remain in one row. None remains None in this mixed-object fixture. The value column is duplicated accordingly. A successful call with no error could mislead us into thinking everything is fine. But by design the row for index 1 still appears (with NaN) even though “there was no element to expand”. Our job is to now compare this output to the expected manifest.
6. Run the comparison case
Compare each actual result with the expected manifest. Use pandas.testing.assert_frame_equal (or equivalent) to check equality. For instance:
import pandas.testing as tm
expected = pd.DataFrame({
'id': [0,0,1,2,3],
'list_col': [10, 20, float('nan'), 'foo', None],
'value': [1, 1, 2, 3, 4]
})
tm.assert_frame_equal(
result.reset_index(drop=True), expected, check_dtype=False
)We reset index to align row labels (or set ignore_index=True in explode) because the default index duplication does not matter for value equality. We allow flexibility in dtype with check_dtype=False; this option relaxes dtype comparison, not the meaning of missing values. The expected data preserves None separately from the NaN placeholder. Index provenance is checked separately below. If this assertion passes, the output exactly matches our contract. If it fails, we have a discrepancy (mismatched rows or values).
Detect missing or duplicated evidence: We also check that every input row is accounted for. For our fixture, counting only list elements plus scalar rows gives 2 (from id=0) + 0 (id=1) + 1 (id=2) + 1 (id=3) = 4 rows before the empty-list placeholder is included. The documented output has 5 rows because the empty list contributes one additional NaN row. We verify that only those extra rows are the ones mandated by pandas (i.e. one per empty list) and no unexpected duplicates. If an extra row appeared where it shouldn’t, or an expected row is missing, that would trigger a failure. (For example, if a non-list row had somehow expanded erroneously, it would show up as duplicate evidence.)
We ensure consistency: e.g. the number of output rows should equal
num_rows = sum(
1 if not isinstance(x, list) else max(len(x), 1)
for x in df_fixture['list_col']
)which in this case is 2 + 1 + 1 + 1 = 5. The two-element list contributes two rows; the empty list, scalar string, and None each contribute one. This expected count includes the documented NaN placeholder for the empty list.
7. Test negative and boundary controls
We must also test error conditions and unusual inputs. For instance, exploding mismatched lists or empty column spec should raise ValueError as documented.
Mismatched lengths:
df_err = pd.DataFrame({'A': [[1,2], [3]], 'B': [[1], []]})
# EXPECTED / NOT EXECUTED: mismatched lengths must raise ValueError.
try:
df_err.explode(['A','B'])
except ValueError:
print("ValueError as expected")
else:
raise AssertionError("Expected ValueError not raised")Here, A[0] has length 2 while B[0] has length 1. This first-row mismatch should cause a ValueError. If it does, we PASS the negative test. If it does not (i.e. pandas silently does something), that violates the contract.
Empty explode list:
try:
df_fixture.explode([]) # no columns specified
except ValueError:
print("ValueError for empty column list as expected")
else:
raise AssertionError("Expected ValueError for empty list not raised")Pandas docs state that providing an empty list for columns triggers a ValueError. We verify this behavior.
Duplicate column names: If the DataFrame has non-unique column names and we attempt explode, pandas also raises a ValueError. (We can create such a case to confirm if needed.)
These negative/boundary tests ensure that pandas is enforcing its documented preconditions. We also confirm that when an error occurs, no partial transformation happens: the original DataFrame remains unchanged (pandas explode returns a new object and is not in-place). For instance:
df_copy = df_err.copy()
try:
df_err.explode(['A','B'])
except ValueError:
pass
assert df_err.equals(df_copy) # original must be intact8. Collect per-case raw evidence
Document the results and any logs for each test. In practice we would save snapshots of the DataFrames before and after, as well as any console output. The following ledger describes expected evidence, not observations from an executed run. Every entry remains EXPECTED / NOT EXECUTED:
Fixture explode: Expected output includes the NaN placeholder for the empty list. Record whether id=1 produces NaN and id=3 retains None. All other values should match the inputs.
Scalar and None: Both should behave as scalars (no expansion beyond one row). (The documentation keeps scalars unchanged and produces NaN for empty list-like values.)
Negative control: Expected output equals input, with no changes to row count or values.
Mismatched explode: ValueError is expected. Record the stack trace or exception message after execution.
Error state: After a ValueError on the mismatched case, verify that the DataFrame remains unmodified: the row count should still be 2, and no partial output should be returned. Record that check as evidence of rerun safety.
Any discrepancy between expected and observed for each case is highlighted. In our example, the expected “misleading success” is that the empty-list row remains present, as documented by pandas. We record this explicitly.
Check the actual state after failure
If any explode call fails, we verify no side effects. For example, after the mismatched-lists test above, df_err should still have its original shape. This ensures we can safely retry or recover.
9. Reconcile identities and counts
Verify that each output row can be traced to an input row (row provenance) and that totals match. This is the heart of the audit. For each test:
Row counts: Compute the expected number of rows (as above) and compare to the actual output length. If the contract says “one row per empty list” and we see exactly one such row, we’re good. If not, that’s an error.
Index provenance: Use reset_index() to attach the original index into the output, or keep the original index if ignore_index=False. For example:
out = result.reset_index()Now out['index'] retains the original index label. Where labels are duplicated, the fixture’s unique id identifies the source row. For each source id, check that the output count equals the list length for a nonempty list and one for an empty list or scalar. The official pandas doc example shows that exploding preserves the original index values.
Reconciliation formula: Similar to the GroupBy ledger method, we can form an identity check: Expected totals must account for the number of output rows contributed by each source row. For instance, the GroupBy article kept a “reconciliation equation of sums”. Here, we ensure that summing a counter column yields consistency. In our fixture, value sums to 1 + 2 + 3 + 4 = 10 before expansion. Its expected total afterward is 1 + 1 + 2 + 3 + 4 = 11, because the value of 1 is copied onto both rows from id=0. Missingness in list_col does not remove the nonmissing values in value from that sum. Check sum_after against this multiplicity-aware expected total, alongside source-id coverage; equality with the unexpanded total is not the right invariant.
If all traces line up, we have confidence that every output row is valid and we didn’t “invent” or “lose” data. (Any missing or extra ID would be immediately spotted in this step.)
10. Explain documented behavior and limits
Summarize what pandas promises about explode and the edge-case behavior the fixture is designed to verify. According to the official docs:
List-like values: Any list/tuple/ndarray in a cell is exploded into multiple rows, with other columns duplicated.
Scalars: Values that are not list-like (including strings, ints, floats, None) are treated as length-1 “lists” and are returned unchanged. This means missing values also remain as single-row entries.
Empty lists: By definition, they have “no elements,” yet pandas chooses to output a row with NaN in that column. This behavior is documented (and visible in examples), though it can be counter-intuitive if one expected the row to disappear. The presence of that NaN row is pandas’ intended behavior, not a bug, so it aligns with the library’s contract.
Index replication: The original index labels are reused for exploded rows. For example, exploding row 0 into 3 parts yields three rows all with index 0. This preserves index labels; use the unique source id to distinguish input rows that share a label.
Multiple columns: Exploding multiple columns at once requires that each column has list-like entries of the same length per row; otherwise a ValueError is raised. If lists match in length, the exploded rows take corresponding elements from each column.
Empty columns argument: Calling explode([]) (no columns specified) is an explicit error, as is attempting to explode a non-existent column.
Exploding sets: The docs note that exploding sets may yield rows in arbitrary order, since sets are inherently unordered. This nondeterminism is a documented limit. (It does not affect our test data because we did not use sets.)
The expected behaviors above follow the documented contracts. Execution is still required to determine whether the selected environment matches them. Emitting NaN for an empty list preserves its source row; any quarantine policy belongs to the surrounding data workflow. The audit principle remains: do not silently discard data.
Bound the repair to the contract
Any fix we apply should adhere to the defined contract. For example, if we had decided (by business rule) that rows from empty lists should be dropped instead of returned as NaN, that rule itself is a change to the contract. According to the data workflow, such invalid records should be “quarantined with issue codes rather than silently discarded”. Thus, if our contract changes, we should document it (e.g. by filtering out those rows or flagging them). But because the pandas contract (as documented) is to output a NaN row, no repair is strictly required here unless our specific policy says otherwise.
11. Apply the smallest safe repair
If, for some reason, the NaN row from an empty list is undesirable (for example, if the business rule is “completely drop empty entries”), we implement the minimal change to the output. If the policy is to remove all missing exploded values, a simple fix is to drop rows where the exploded column is null. This is broader than removing only rows that originated from empty lists. For instance:
if drop_missing_values:
repaired = result.dropna(subset=['list_col'])
else:
repaired = resultThis removes both the empty-list NaN placeholder and the original None row. To exclude only empty lists, identify those source rows before expansion instead. We then reset the index if desired:
repaired = repaired.reset_index(drop=True)We treat this as a safe, incremental fix, applied only if it accords with the contract. After this repair, we should re-run the assertions: the expected row count is 3, and value should sum to 5. Reconcile the excluded source ids and totals against the approved filtering policy. (Note: dropping rows silently might conflict with the “no silent discard” principle unless we log or flag it. So in production we might add an explicit flag column instead of truly dropping.)
12. Test rollback and rerun safety
Finally, verify that the process can be safely rerun or rolled back. This means: if a fix fails or if software is updated, we can start over with the original raw data. Confirm that test changes do not persist to the source DataFrame when working on copies. For example, after filtering missing values into repaired, the original df_fixture should remain intact. This is by design: explode and dropna return new DataFrames.
If a newer pandas version is used later (say pandas 3.1), rerun the entire fixture to ensure behavior is unchanged. Indeed, as the Shapely playbook notes, changing library versions creates a new test condition: one should “generate a new observation record” for the new version. We freeze the environment, so a version upgrade triggers a full retest.
Assign operational ownership
Document who is responsible for this check. Typically, the data engineering or QA team owning this pipeline maintains the fixture and scripts. The owner must ensure the pinned environment (versions) is stored (e.g. in a requirements.txt) and that this test suite is run whenever pandas or related libraries are updated. If any check fails, the owner decides on ACCEPT/REPAIR/RERUN/HOLD as below.
13. Decide ACCEPT, REPAIR, RERUN or HOLD
We evaluate final outcomes against the admission contract:
ACCEPT: If all output matches the expected manifest, and the behavior (such as the NaN row for empty list) aligns with our defined contract and pandas docs, we ACCEPT this change. For this fixture, a NaN placeholder for the empty list is expected. Accept the run only after the recorded output and every required control match the manifest.
REPAIR: If the output violates the contract in an implementable way, we REPAIR. For instance, if our policy were to omit empty-list rows, we would identify and filter those source rows; the broader dropna example also removes other missing values. That change (if agreed upon) would be tested and then accepted.
RERUN: If the discrepancy is due to environment (different pandas version, etc.), we mark RERUN after aligning versions. For example, an upgrade from pandas 2.x to pandas 3.x creates a new test condition and requires a new evidence record, even when the documented empty-list behavior is unchanged.
HOLD: If there is ambiguity or a third-party dependency issue that we cannot immediately fix or reproduce, we HOLD for further analysis. For example, if explode started raising an unexpected error only on certain data, we might pause and escalate.
The examples here remain EXPECTED / NOT EXECUTED, so the release decision is HOLD until the pinned fixture produces complete evidence and all required comparisons pass. The documented empty-list NaN placeholder can satisfy the contract without repair. If the data contract excludes missing rows, filtering is a REPAIR action that requires an explicit policy, reconciliation, and another verified run.
14. Develop related skills with Refonte Learning
Data engineering and testing skills are critical for rigorous data pipelines. Refonte Learning offers a Data Engineering Training & Internship Program that covers these topics. Their program is a 3-month course (12–14 hours/week) teaching big data tools and best practices such as Hadoop, Spark, ETL pipelines, and data governance. It includes hands-on projects and a virtual internship so you can apply tests like the above in real scenarios. Graduates earn both a Training Certificate and an Internship Certificate. Consider joining Refonte’s Data Engineering program to deepen your skills in reliable data processing and validation.
For more details, visit Refonte Learning’s Data Engineering Program. It outlines prerequisites (bachelor’s student or higher), curriculum, mentorship, and internship opportunities.