A pandas GroupBy table shows two rows each for North and South and zero for East. East appears only in the declared categories.

pandas GroupBy Included an Empty Category. Was There a Source Row?

Sat, Nov 21, 2026

A green pandas command can produce exactly the result that creates an operational ambiguity: a categorical GroupBy output contains a category, its group size is zero, and no source row ever carried that category value. That is not necessarily corruption. With categorical groupers, observed=False can materialize unused categories in the result, while dropna independently controls whether rows whose grouping keys are missing participate. In pandas 2.2.3, observed defaulted to False; current pandas 3.1.1 documentation shows the default changed to True in pandas 3.0.0.

The release question is therefore not “did groupby() succeed?” It is: for every output group, can we state whether one or more source row identities actually belonged to that group under the declared categorical and missing-key policy? DataFrameGroupBy.size() reports rows in groups, but an output label can exist because the categorical domain was expanded, not because a row produced it.

This playbook constrains the diagnosis to six authored cases: one unused category, an observed=True comparison, a dropna toggle, two categorical keys, empty-count-versus-membership reconciliation, and category reordering with stable row IDs. The fixture is synthetic and disposable. No production access is required. The expected manifest is written independently of the grouping output, the failing inference is preserved as a negative control, and release decisions are limited to ACCEPT, REPAIR, RERUN, or HOLD.

A local execution for this research was performed on CPython 3.13.5 and pandas 2.2.3 using the harness below. All six authored cases and the deliberate negative control behaved as expected. Those observations qualify only that environment; the current pandas 3.1.1 documentation is used separately to establish the changed default and present-day API boundary.

Define the Exact Operational Acceptance Contract

The acceptance contract must describe row admission, not output cosmetics.

For a categorical grouping key, distinguish three objects: the categorical domain, the observed source values, and the groups emitted by pandas. A Categorical can contain declared categories that no row uses. Pandas documents categorical data as values backed by a categories array and integer codes; missing values are not categories, and unused categories can remain in the categorical domain.

For this playbook, acceptance means all of the following are evidenced simultaneously:

  • Every input row has a unique, preserved row_id.

  • The categorical domains are captured before grouping.

  • observed and dropna are explicit rather than inherited silently from defaults.

  • Every non-empty output group can be mapped to the exact admitted row_id set.

  • Every zero-sized categorical group is classified as domain-only unless source membership proves otherwise.

  • Missing-key rows are either included or excluded exactly as the authored dropna policy specifies.

  • Reordering categories may change category ordering, but may not silently change semantic row identity.

  • The runtime identity, package version, fixture identity, execution time, raw result and expected manifest are retained together.

Refonte Learning’s guide to missing GroupBy keys shows which rows disappear when dropna=True with nullable string grouping columns. Those keys are non-categorical, so observed plays no role in expanding unseen categories. With categorical keys, the question is whether an emitted group has source-row lineage, including when an unused category produces a group of size zero.

A zero group is accepted only when the manifest authorizes domain expansion and the lineage oracle independently says the member-ID set is empty. A non-zero group is accepted only when its count equals the cardinality of the independently derived row-ID set. If either statement cannot be proved, the release does not pass merely because pandas returned successfully.

Document the Runtime and Version Boundary

Version boundaries matter because observed changed default behavior.

The pandas 2.2.3 DataFrame.groupby documentation defines observed=False as the default for categorical groupers and says observed=True shows only observed values, while observed=False shows all categorical values. That documentation also marks the default for future change. Current pandas 3.1.1 documentation shows observed=True as the default and states that this default changed in version 3.0.0.

dropna is a separate switch. Both the 2.2.3 and 3.1.1 API documentation describe dropna=True as excluding rows whose grouping keys contain NA values and dropna=False as treating NA values as group keys. Do not infer dropna behavior from observed, or vice versa.

The test boundary used here is:

Boundary item

Qualified value

Python implementation

CPython

Python version actually executed

3.13.5

pandas version actually executed

2.2.3

Fixture

five synthetic source rows

Categorical domains

region: North, South, East; channel: web, store, partner

Execution scope

local disposable process only

Current-doc comparison

pandas 3.1.1

Production data

none

In pandas 2.2.3, pandas.core.groupby.DataFrameGroupBy.size reports the number of rows in each group. The pandas 3.1.1 API documents the same operation as pandas.api.typing.DataFrameGroupBy.size.

Pin and Record the Environment

The measurable environment check is not “import succeeded.” It is exact identity agreement.

The reproducible target for the harness is CPython 3.13.5 plus pandas 2.2.3, because that pair was available and actually executed during this research. The pandas pin corresponds to version-specific documentation in which observed=False was still the default.

python3.13 --version
python3.13 -m venv .venv-groupby-lineage
. .venv-groupby-lineage/bin/activate
python -m pip install --upgrade pip
python -m pip install pandas==2.2.3
python -c "import sys,pandas as pd; print(sys.version); print(pd.__version__)"
python pandas_groupby_empty_category_fixture.py \
  > pandas-groupby-empty-category-evidence.json
printf 'exit_status=%s\n' "$?"

Independent expectation: Python must identify the intended interpreter, pandas must report 2.2.3, and the harness must record those values in the evidence artifact before any result is admissible.

Rejection criterion: a different pandas version, an unrecorded version, a missing evidence artifact, or an environment whose result is being compared to the wrong versioned documentation means RERUN, not ACCEPT. In particular, a run on pandas 3.x must not be interpreted using the 2.2.3 default for observed.

Create a Disposable Controlled Fixture

The fixture should make “category exists” and “source row exists” visibly different.

Use five rows. East is declared in the region categorical but never appears in region_raw. partner is declared in the channel categorical but never appears in channel_raw. One row has a missing region and one has a missing channel. The fixture therefore exercises unused categories and missing group keys without conflating them.

rows = [
    ("R1", "North", "web",   10),
    ("R2", "North", "store", 20),
    ("R3", "South", "web",   30),
    ("R4", None,    "web",   40),
    ("R5", "South", None,    50),
]

df = pd.DataFrame(
    rows,
    columns=["row_id", "region_raw", "channel_raw", "amount"],
)

df["region"] = pd.Categorical(
    df["region_raw"],
    categories=["North", "South", "East"],
    ordered=True,
)
df["channel"] = pd.Categorical(
    df["channel_raw"],
    categories=["web", "store", "partner"],
    ordered=True,
)

assert df["row_id"].is_unique

Pandas documents that explicitly supplied categorical categories need not all occur in the values, and the categorical guide demonstrates that unused categories remain available until explicitly removed. That documented model is the mechanism the fixture is designed to expose.

Refonte Learning’s guide to auditing rows in pairwise correlations uses source IDs to identify which rows contribute to each coefficient. Different column pairs can use different intersections of rows. In categorical grouping, the output can include a group whose set of source IDs is empty.

Fixture teardown is deliberately trivial:

deactivate 2>/dev/null || true
rm -rf .venv-groupby-lineage
rm -f pandas_groupby_empty_category_fixture.py
rm -f pandas-groupby-empty-category-evidence.json

Do not run teardown until the evidence artifact has been copied to the review package when evidence retention is required.

Author an Independent Expected-Case Manifest

The expected manifest must exist before looking at the pandas results. Otherwise the test risks becoming circular: “pandas produced X, therefore X must have been expected.”

The six required cases are:

Case

Independently authored expectation

Initial status

C1 unused category included

observed=False, dropna=True emits North=2, South=2, East=0

EXPECTED / NOT EXECUTED before run

C2 observed=True comparison

North=2 and South=2; East absent

EXPECTED / NOT EXECUTED before run

C3 missing key, dropna toggled

dropna=True counts four rows; dropna=False counts five and NA group has one row

EXPECTED / NOT EXECUTED before run

C4 two categorical keys

3 region categories × 3 channel categories = 9 result combinations under observed=False, dropna=True; three occupied, six empty

EXPECTED / NOT EXECUTED before run

C5 empty count versus membership

East appears in output with size 0, while its source-ID tuple is empty

EXPECTED / NOT EXECUTED before run

C6 category reorder

row-ID-to-semantic-region mapping and counts by label remain unchanged after reordering categories

EXPECTED / NOT EXECUTED before run

These expectations were authored from the literal fixture and categorical domains, not copied from observed output. They are consistent with pandas documentation that observed=False exposes unused categorical values and with GroupBy.size() being a row count.

The manifest deliberately avoids requiring category codes to remain stable. Pandas describes categoricals as category arrays plus integer codes, and reordering the category order is an operation on the categorical metadata. Source lineage therefore uses row_id plus semantic source values, not code positions.

Preserve a Failing Case Before Repair

The misleading rule is:

source_row_exists = group_label in grouped.index

For East, that rule predicts a source row because East is present in the grouped index when observed=False. The fixture independently says there is no source row_id with region_raw == "East".

The negative control preserves that false rule deliberately:

source_east_ids = tuple(
    df.loc[df["region_raw"].eq("East"), "row_id"]
)

negative_control_rejected = False
try:
    assert ("East" in grouped.index) == bool(source_east_ids)
except AssertionError:
    negative_control_rejected = True

assert negative_control_rejected

Independent expectation: the assertion must fail, proving that output-label presence is not a valid source-lineage oracle.

Rejection criterion: if this control unexpectedly passes, either the fixture has changed, a source row now genuinely contains East, the grouped object no longer contains the unused category, or the negative control is no longer exercising the intended failure. Any of those conditions requires HOLD until the fixture and manifest are reconciled.

Reproduce the Misleading Success Signal

The smallest misleading success is an entirely successful GroupBy.size():

grouped = (
    df.groupby(
        "region",
        observed=False,
        dropna=True,
    )
    .size()
)
print(grouped)

For the executed pandas 2.2.3 fixture, the observed result was semantically:

North    2
South    2
East     0

This matched C1. The command completed normally, and East was present even though no source row had region_raw == "East". That observation is consistent with pandas’ categorical guide, which explicitly shows unused categories in groupby results under observed=False.

The success signal is misleading only if an operator assigns it more meaning than pandas documents. pandas is reporting a category from the categorical domain with a group size of zero. It is not claiming that a source row carried that value. GroupBy.size() documents the number of rows in each group; zero is therefore meaningful evidence of no rows in that group under the executed grouping policy, not evidence that the label originated in a source record.

That distinction is operationally decisive. A downstream report may intentionally require a complete category domain, including zeros. In that contract, East=0 could be exactly correct. A lineage report asking “which source rows generated each group?” must answer differently: East has no source members in this fixture.

This is why a process exit code of zero is only execution evidence. It cannot decide admission. The release reviewer needs both result-state evidence and source-identity evidence.

Run a Correctness Comparison

The controlled positive comparison changes exactly one relevant grouping parameter:

observed_only = (
    df.groupby(
        "region",
        observed=True,
        dropna=True,
    )
    .size()
)

The executed pandas 2.2.3 result matched the independently authored C2 expectation: North=2, South=2, and no East output row.

That result is directly predicted by pandas’ documented observed semantics: when a grouper is categorical, True shows observed values, whereas False shows all categorical values. The comparison is useful because it changes the disputed mechanism while holding the source rows, categorical dtype, dropna policy and aggregation constant.

Do not treat this as a recommendation that every workload should use observed=True. If the reporting contract requires every category in the domain, observed=False may be intentional. The acceptance rule is not “fewer groups are better”; it is “group emission matches the declared policy and group lineage is not inferred from label presence.”

Refonte Learning’s guide to tracing zeros after reindexing distinguishes source labels from labels introduced to satisfy a target index. It uses that distinction to identify structurally introduced zeros. In the GroupBy fixture, the categorical domain itself can supply an output label without a call to reindex() or an explicit target index.

Reject an Invalid Negative Control

A useful negative control must fail for the same semantic reason that worries us. Changing an unrelated measure, such as amount, would be a weak control because GroupBy.size() counts rows rather than non-null values in a measure column.

The valid negative control is therefore the false lineage inference:

bad_rule = "East" in grouped.index
real_source_membership = bool(
    tuple(df.loc[df["region_raw"].eq("East"), "row_id"])
)

assert bad_rule == real_source_membership

The executed control raised AssertionError, as independently expected.

Expected failure: bad_rule is True, while real_source_membership is False.

Rejection criterion: do not accept a “negative control” that would pass regardless of categorical expansion, that changes only unrelated data, or that tests the same expression twice. A control that cannot discriminate the defect is not evidence.

The controlled positive case and this controlled failure now bracket the diagnosis: observed=True removes the unused category; observed=False emits it; and the source membership query remains empty. That triangulation is stronger than simply seeing East=0.

Test Negative and Boundary Conditions

The remaining boundary tests establish that unused categories and missing keys are orthogonal mechanisms.

For C3, execute the same categorical region grouper twice:

drop = df.groupby(
    "region", observed=True, dropna=True
).size()

keep = df.groupby(
    "region", observed=True, dropna=False
).size()

The authored expectation was four rows counted with dropna=True, versus all five rows with dropna=False, with one row in the NA group. The executed pandas 2.2.3 result matched. pandas’ groupby guide states that grouping-key NA values are excluded by default and can be included by setting dropna=False.

For C4, group by both categoricals:

two_keys = df.groupby(
    ["region", "channel"],
    observed=False,
    dropna=True,
).size()

The authored domain has three region categories and three channel categories, giving nine category combinations. Only R1, R2, and R3 have both grouping keys present, so the oracle expects total size three across nine emitted combinations, with six zero-size combinations. The executed result matched exactly.

This case is important because an empty group need not correspond merely to one unused single category. It can arise from an unobserved combination of categorical values. The pandas categorical guide shows multi-key grouping with observed=False producing combinations for an unused categorical value.

A release reviewer should therefore never infer source membership from Cartesian domain membership. For a two-key result such as (East, partner), the correct question is which source IDs satisfy both semantic key predicates under the missing-key policy. In this fixture, the answer is none.

Capture Raw Evidence and Exit Status

Evidence should make later interpretation possible without rerunning the code immediately. At minimum, preserve environment identity, execution time, fixture size, expected state, observed state, pass/fail status and a human-readable explanation.

Save the complete harness as pandas_groupby_empty_category_fixture.py and run it with the environment commands above.

import json
import platform
import sys
from datetime import datetime, timezone

import pandas as pd

PINNED_PANDAS = "2.2.3"
assert pd.__version__ == PINNED_PANDAS, (
    pd.__version__,
    PINNED_PANDAS,
)

ROWS = [
    ("R1", "North", "web",   10),
    ("R2", "North", "store", 20),
    ("R3", "South", "web",   30),
    ("R4", None,    "web",   40),
    ("R5", "South", None,    50),
]

df = pd.DataFrame(
    ROWS,
    columns=[
        "row_id",
        "region_raw",
        "channel_raw",
        "amount",
    ],
)

df["region"] = pd.Categorical(
    df["region_raw"],
    categories=["North", "South", "East"],
    ordered=True,
)
df["channel"] = pd.Categorical(
    df["channel_raw"],
    categories=["web", "store", "partner"],
    ordered=True,
)

assert df["row_id"].is_unique

ledger = []

def record(case, expected, observed, passed, detail):
    ledger.append(
        {
            "case": case,
            "expected": expected,
            "observed": observed,
            "passed": bool(passed),
            "detail": detail,
        }
    )

# C1: unused category included.
s1 = df.groupby(
    "region",
    observed=False,
    dropna=True,
).size()
obs1 = {str(k): int(v) for k, v in s1.items()}
exp1 = {"North": 2, "South": 2, "East": 0}
record(
    "C1_unused_category_included",
    exp1,
    obs1,
    obs1 == exp1,
    "observed=False, dropna=True",
)

# C2: observed=True comparison.
s2 = df.groupby(
    "region",
    observed=True,
    dropna=True,
).size()
obs2 = {str(k): int(v) for k, v in s2.items()}
exp2 = {"North": 2, "South": 2}
record(
    "C2_observed_true",
    exp2,
    obs2,
    obs2 == exp2,
    "unused East omitted",
)

# C3: missing key with dropna toggled.
s3a = df.groupby(
    "region",
    observed=True,
    dropna=True,
).size()
s3b = df.groupby(
    "region",
    observed=True,
    dropna=False,
).size()

obs3 = {
    "dropna_true_total": int(s3a.sum()),
    "dropna_false_total": int(s3b.sum()),
    "dropna_false_has_na": bool(
        s3b.index.isna().any()
    ),
    "dropna_false_na_count": (
        int(s3b[s3b.index.isna()].iloc[0])
        if s3b.index.isna().any()
        else None
    ),
}
exp3 = {
    "dropna_true_total": 4,
    "dropna_false_total": 5,
    "dropna_false_has_na": True,
    "dropna_false_na_count": 1,
}
record(
    "C3_dropna_toggle",
    exp3,
    obs3,
    obs3 == exp3,
    "R4 has missing region",
)

# C4: two categorical grouping keys.
s4 = df.groupby(
    ["region", "channel"],
    observed=False,
    dropna=True,
).size()

obs4 = {
    "group_count": int(len(s4)),
    "total_rows_counted": int(s4.sum()),
    "zero_groups": int((s4 == 0).sum()),
}
exp4 = {
    "group_count": 9,
    "total_rows_counted": 3,
    "zero_groups": 6,
}
record(
    "C4_two_categorical_keys",
    exp4,
    obs4,
    obs4 == exp4,
    "3x3 domain; missing-key rows excluded",
)

# C5: empty count versus source membership.
source_east_ids = tuple(
    df.loc[
        df["region_raw"].eq("East"),
        "row_id",
    ]
)
obs5 = {
    "east_in_output": bool("East" in s1.index),
    "east_count": int(s1.loc["East"]),
    "east_source_ids": source_east_ids,
}
exp5 = {
    "east_in_output": True,
    "east_count": 0,
    "east_source_ids": (),
}
record(
    "C5_empty_count_vs_lineage",
    exp5,
    obs5,
    obs5 == exp5,
    "output label is not source membership",
)

# C6: reorder categories; preserve source identity.
before_identity = tuple(
    df[["row_id", "region_raw"]]
    .itertuples(index=False, name=None)
)

reordered = df.copy()
reordered["region"] = (
    reordered["region"]
    .cat.reorder_categories(
        ["East", "South", "North"],
        ordered=True,
    )
)

after_identity = tuple(
    reordered[["row_id", "region_raw"]]
    .itertuples(index=False, name=None)
)

before_counts = {
    str(k): int(v)
    for k, v in df.groupby(
        "region",
        observed=False,
        dropna=True,
    ).size().items()
}
after_counts = {
    str(k): int(v)
    for k, v in reordered.groupby(
        "region",
        observed=False,
        dropna=True,
    ).size().items()
}

obs6 = {
    "identity_unchanged":
        before_identity == after_identity,
    "counts_by_label_equal":
        before_counts == after_counts,
    "new_category_order":
        list(reordered["region"].cat.categories),
}
exp6 = {
    "identity_unchanged": True,
    "counts_by_label_equal": True,
    "new_category_order":
        ["East", "South", "North"],
}
record(
    "C6_reorder_categories_stable_ids",
    exp6,
    obs6,
    obs6 == exp6,
    "semantic IDs, not category codes, are lineage",
)

# Deliberately false lineage inference.
negative_control_rejected = False
try:
    assert (
        ("East" in s1.index)
        == bool(source_east_ids)
    )
except AssertionError:
    negative_control_rejected = True

record(
    "NC_output_presence_proves_lineage",
    "AssertionError (control rejected)",
    (
        "AssertionError"
        if negative_control_rejected
        else "no error"
    ),
    negative_control_rejected,
    "deliberately false lineage rule",
)

print(
    json.dumps(
        {
            "timestamp_utc":
                datetime.now(
                    timezone.utc
                ).isoformat(),
            "python":
                sys.version.split()[0],
            "implementation":
                platform.python_implementation(),
            "pandas":
                pd.__version__,
            "fixture_rows":
                len(df),
            "ledger":
                ledger,
            "all_passed":
                all(x["passed"] for x in ledger),
        },
        indent=2,
        default=list,
    )
)

The research execution produced UTC timestamp 2026-11-21T07:24:09.943054+00:00, CPython 3.13.5, pandas 2.2.3, five fixture rows, six passing authored cases, and a successfully rejected negative control. These are local observations from this research run, not pandas-wide guarantees.

Detect Missing Evidence and Duplicates

Before accepting the ledger, validate the ledger itself.

required = {
    "C1_unused_category_included",
    "C2_observed_true",
    "C3_dropna_toggle",
    "C4_two_categorical_keys",
    "C5_empty_count_vs_lineage",
    "C6_reorder_categories_stable_ids",
    "NC_output_presence_proves_lineage",
}

actual = [x["case"] for x in ledger]

assert len(actual) == len(set(actual)), \
    "duplicate evidence case"
assert set(actual) == required, \
    "missing or unexpected evidence case"
assert df["row_id"].is_unique, \
    "source identity is ambiguous"

Independent expectation: the ledger contains exactly seven entries (six manifest cases and one negative control), and every source row ID is unique.

Rejection criterion: missing case evidence, duplicate case names, duplicate row identities, or an unreadable evidence artifact means RERUN when the source is still trustworthy and the run can be reconstructed; use HOLD if source identity itself is ambiguous.

Reconcile Observed State Against Source Identity

The key reconciliation is not merely:

sum(group sizes) == number of input rows

That equality is conditional on dropna, because dropna=True intentionally excludes rows with missing group keys. The stronger ledger records the policy and membership for every group.

For one categorical key, build the lineage oracle from the preserved semantic source column:

def source_ids_for_region(label):
    return tuple(
        df.loc[
            df["region_raw"].eq(label),
            "row_id",
        ]
    )

Then compare it to the grouped size:

for label in ["North", "South", "East"]:
    ids = source_ids_for_region(label)
    assert s1.loc[label] == len(ids)

For East, that assertion is 0 == len(()). The output row is legitimate, but its lineage is explicitly empty.

For missing-key reconciliation, the source oracle is similarly direct:

missing_region_ids = tuple(
    df.loc[
        df["region_raw"].isna(),
        "row_id",
    ]
)
assert missing_region_ids == ("R4",)

Under dropna=False, the NA group must contain one row; under dropna=True, that row is not admitted to any region group. pandas documents this missing-key behavior directly.

This source-identity discipline resembles the operational reasoning in Refonte Learning’s PostgreSQL identity-return validation, where a command can complete without error yet still fail the caller’s identity contract. The technologies and failure mechanisms differ, but the reusable engineering principle is relevant: successful execution is not identical to satisfying an identity-bearing acceptance contract.

The pandas decision remains narrower: does each emitted group correspond to zero or more explicitly identified source rows exactly as the grouping policy says?

Separate Official Guarantees From Local Observations

Keep four evidence classes separate.

Documented semantics. pandas documents that categorical groupers use observed to choose observed-only versus all categorical values, that dropna controls missing grouping keys, and that GroupBy.size() reports rows per group. pandas’ categorical guide also demonstrates unused categories appearing in grouped results under observed=False.

Independently authored expectations. The six-case manifest came from the literal five-row fixture and declared category domains. It is not a statement by the pandas project.

Local observations. On CPython 3.13.5 with pandas 2.2.3, every manifest case matched, and the false “output label proves source membership” assertion failed as intended. That is evidence for that exact local run only.

Inference. Because C1, C2 and C5 jointly show that East appears only when unused categorical values are emitted and has an empty source-ID set, the reasonable fixture-level inference is that East is a domain-derived empty group rather than a row-derived group. That inference is supported by the documented categorical semantics and the local source-membership evidence.

Do not promote any local observation to a guarantee for every pandas version. The default change in pandas 3.0.0 is exactly why that would be unsafe.

Bound the Scope of the Fix

The repair target must be no larger than the failed contract.

If the report should include only source-observed categorical values, the narrow change is explicit observed=True:

result = df.groupby(
    "region",
    observed=True,
    dropna=declared_dropna_policy,
).size()

If the report intentionally requires the complete categorical domain, retain observed=False and repair the downstream lineage interpretation instead:

result = df.groupby(
    "region",
    observed=False,
    dropna=declared_dropna_policy,
).size()

Measurable test: rerun all six manifest cases after the code change.

Independent expectation: only the behavior explicitly authorized by the repair changes. Source IDs, source values and unrelated missing-key policy remain invariant.

Rejection criterion: if a proposed fix removes categories, fills missing keys, rewrites row IDs, changes unrelated measures, or otherwise alters more than the declared grouping-policy boundary, classify it as HOLD pending a new contract rather than approving scope creep.

Implement a Narrow Repair or Safer Workflow

There are two legitimate repairs because there are two legitimate contracts.

For an observed-only report, make observed=True explicit even on versions where it is already the default:

grouped = (
    df.groupby(
        ["region", "channel"],
        observed=True,
        dropna=False,
    )
    .size()
)

Explicitness protects the review from version-default ambiguity. On pandas 2.2.3 the default was False; on pandas 3.1.1 the default is True.

For a complete-domain report, keep observed=False, but emit provenance fields alongside counts:

domain_counts = df.groupby(
    "region",
    observed=False,
    dropna=True,
).size()

lineage = {
    label: tuple(
        df.loc[
            df["region_raw"].eq(label),
            "row_id",
        ]
    )
    for label in df["region"].cat.categories
}

Then classify:

state = {
    label: {
        "rows": int(domain_counts.loc[label]),
        "row_ids": lineage[label],
        "origin": (
            "source-observed"
            if lineage[label]
            else "categorical-domain-only"
        ),
    }
    for label in domain_counts.index
}

Do not “repair” an unused category by inventing a synthetic source row. A structural zero in a report and a source record with value zero are different provenance states. Likewise, do not call remove_unused_categories() merely to make the symptom disappear unless the category domain itself is intended to shrink. Pandas exposes that operation specifically to remove unused categories; using it changes the categorical domain and is therefore a schema decision, not merely presentation cleanup.

The safer workflow is to keep domain policy and source-lineage policy explicit and separate.

Test Recovery, Rollback and Rerun Safety

A repair is not ready merely because its preferred branch now passes. Recovery must demonstrate that the source fixture can be restored, the original failing behavior can still be reproduced, and reruns do not mutate source identity.

Use a preserved copy:

baseline = df.copy(deep=True)

# Candidate operation.
candidate = baseline.groupby(
    "region",
    observed=True,
    dropna=True,
).size()

# Rollback validation.
pd.testing.assert_frame_equal(df, baseline)
assert tuple(df["row_id"]) == (
    "R1", "R2", "R3", "R4", "R5"
)

Then reproduce the original categorical-domain branch again:

reproduced = baseline.groupby(
    "region",
    observed=False,
    dropna=True,
).size()

assert reproduced.loc["East"] == 0

That is rollback safety at fixture scale: the repair did not destroy the ability to reconstruct the original state.

C6 adds an important category-order boundary. Series.cat.reorder_categories() reorders the categorical categories while retaining the displayed values; pandas’ categorical accessor documentation demonstrates that the category order changes while values remain the same. In the executed fixture, reordering ["North","South","East"] to ["East","South","North"] preserved the (row_id, region_raw) sequence and counts by semantic label.

This resembles the evidence discipline in Refonte Learning’s Python task context snapshot validation, which also separates a green operation from the identity state captured by the operation and uses pinned controls rather than trusting execution alone. The pandas repair remains specifically about row lineage and categorical group emission.

Assign Ownership and Escalation

Assign ownership by evidence class rather than by whoever happens to run the command.

The code owner owns explicit observed and dropna configuration. The data-contract owner owns whether an unused category should appear in a released report. The source-data owner owns the uniqueness and semantic meaning of row_id. The release reviewer owns the final comparison of expected manifest, environment identity and evidence ledger.

Measurable test: every failed ledger item must have one owner and one required evidence artifact before rerun.

Independent expectation: no case is accepted from verbal explanation alone; the corrected run must replace or append evidence without overwriting the failing artifact.

Rejection criterion: ambiguous source IDs, disagreement about whether domain-only categories belong in the output, unknown version identity, or a repair that changes the declared data contract without approval requires HOLD. A transient execution or artifact problem with an otherwise agreed contract can be RERUN.

Decide ACCEPT, REPAIR, RERUN or HOLD

The decision must follow the evidence, not the visual appearance of the grouped table.

Evidence state

Decision

Required action

Version, fixture identity, manifest and all applicable cases agree; empty groups have explicit empty lineage; negative control is rejected

ACCEPT

Release only the qualified behavior

Environment and evidence are sound, but grouping configuration disagrees with the approved observed or dropna contract

REPAIR

Make the smallest policy-aligned code change, then rerun all affected cases

Contract is known, but execution evidence is missing, corrupt, incomplete, duplicated, or from the wrong version

RERUN

Recreate disposable fixture and regenerate evidence

Source identity is ambiguous, category-domain policy is disputed, version cannot be established, or proposed repair exceeds scope

HOLD

Resolve authority or contract before another release decision

For the execution performed during this research, the case ledger was:

Case

Expected before execution

Observed on CPython 3.13.5 / pandas 2.2.3

Evidence result

C1 unused category included

North 2, South 2, East 0

matched

PASS

C2 observed=True

North 2, South 2 only

matched

PASS

C3 dropna toggle

totals 4 vs 5; NA group size 1

matched

PASS

C4 two categorical keys

9 groups, 3 admitted rows, 6 zero groups

matched

PASS

C5 empty group lineage

East emitted; count 0; IDs empty

matched

PASS

C6 reorder categories

semantic identities and counts stable

matched

PASS

Negative control

false output-presence lineage rule must fail

AssertionError observed

CONTROL REJECTED

The evidence therefore supports ACCEPT for this synthetic pandas 2.2.3 fixture and its declared policy, not blanket acceptance for another runtime or an unspecified production workload.

Develop the Relevant Skills With Refonte Learning

Refonte Learning’s entry-level Data Engineering program runs for three months and lists a commitment of 12–14 hours per week. Applicants must be pursuing bachelor’s or postgraduate studies. The curriculum includes Introduction to Data Engineering, Data Warehousing and ETL Processes, and Big Data Technologies and Data Pipeline Design, including Hadoop and Spark.

The program page introduces the educational mentors and describes a virtual internship opportunity. It lists a Training Certificate and a Certificate of Internship for successful completion. Topics include data pipelining, real-time processing, transformation, big-data analytics, security and governance.

For structured practice with data pipelines and data engineering workflows, explore Refonte Learning’s Data Engineering program.