Audit the Rows Behind Each pandas Correlation

A correlation table arrives in an analytical handover looking reassuringly ordinary. It is square, symmetric, finite, and accompanied by counts that appear consistent. Before it becomes an input to another analysis, however, a reviewer asks a more operational question: which source records produced each coefficient?

That question matters because pandas.DataFrame.corr() does not require every coefficient to use one shared set of rows. In pandas 2.2.3, Pearson correlation is computed from pairwise complete observations: each column pair can therefore use its own observed-row intersection. min_periods supplies a minimum observation count for each pair, but it does not establish that different pairs used the same record identities.

This article audits that boundary with owned synthetic float64 DataFrames containing three selected features, x, y, and z, plus stable string record IDs. The central decision is not whether pairwise correlation is intrinsically acceptable or whether complete-case selection is intrinsically superior. It is whether the evidence satisfies the declared sample contract.

The workflow therefore records pair memberships, preserves excluded records, builds a separately declared common-cohort candidate when required, and distinguishes computational eligibility from analytical approval. The controls deliberately include a finite pairwise matrix with different source IDs, a complete four-row identity-matrix control, and a twelve-row pair-only identity-matrix counterexample. Matrix appearance is evidence about the numbers; row identity is evidence about the sample.

Define which rows each coefficient is allowed to use

Before reading any coefficient, write down five things: the selected features (x, y, z), the record-identity field, the Pearson method, the missing-observation rule, and the intended sample policy. That declaration turns “run corr()” into a reviewable analytical contract.

Two contracts are legitimate in different contexts. Under pairwise reporting, the x/y coefficient may use every record observed on both x and y, while x/z and y/z use their own intersections. The report should say that the coefficients are pairwise estimates and preserve those memberships. Under a common-cohort requirement, every coefficient must instead be computed from the same selected records that are observed on all three required features.

The library does not decide which contract answers the analytical question. pandas documents the calculation behavior: DataFrame.corr() computes pairwise correlations while excluding missing values, and Pearson, Kendall, and Spearman are documented as using pairwise complete observations. The owner of the analysis still has to decide whether those pair-specific samples are acceptable for the intended downstream use.

That distinction belongs beside broader statistical interpretation and sampling assumptions, not in place of them. This page stays narrower: it asks whether the rows actually used match the sample policy that was declared.

A useful review statement is therefore explicit: “Pearson correlations may be reported pairwise with coefficient-level record membership,” or “all reported coefficients must use one common x/y/z cohort.” Without that sentence, a finite matrix cannot tell the reviewer which rule was intended.

Separate pairwise observations from a shared complete sample

For one coefficient, the membership rule is straightforward. The eligible records for corr(x, y) are the records for which both x and y are non-missing. In set notation, that is the intersection of the observed-record sets for x and y.

A shared three-feature cohort is narrower. It is the intersection of records observed on x, y, and z. That complete set can coincide with every pairwise set, but it does not have to. In ordinary data the pairwise sets may partly overlap, nest inside one another, or differ by only a few IDs. The six-row fixture used here makes the distinction deliberately extreme so the evidence is unmistakable.

pandas 2.2.3 documents DataFrame.corr() as excluding NA/null values pairwise and states that Pearson correlation uses pairwise complete observations. That is the documented selection behavior being audited here.

A count alone loses information. Suppose three pairs each report n=20. That establishes equal cardinality, not equal membership. {A1,…,A20} and {B1,…,B20} have the same size while containing entirely different records. Even substantial overlap cannot be inferred from the counts. A correlation matrix sample-count table is useful, but for a row-identity audit it needs either the IDs themselves or a durable membership artifact from which those IDs can be recovered.

Read min_periods as a condition on each pair

In pandas 2.2.3, min_periods is the minimum number of observations required per pair of columns for a valid Pearson or Spearman result. It is therefore a coefficient-eligibility threshold, not a complete-case selector.

Three separate questions should remain separate:

1. Are the relevant values missing or present?

2. Does this particular pair meet the required observation count?

3. Do both variables vary enough for Pearson correlation to be defined?

Passing a pair-count floor does not answer the third question, and raising the floor does not manufacture a common sample. Likewise, a count chosen for a teaching control does not establish power, representativeness, or inferential adequacy. In this laboratory, min_periods is used only to demonstrate computation boundaries.

Prepare the three-feature laboratory

The reference execution supplied for this article used Python 3.12.14, pandas 2.2.3, NumPy 2.3.5, and Linux 6.18.57-x86_64-with-glibc2.39. The three analytical columns are explicitly constructed as float64. Readers who need prerequisite context on the surrounding ecosystem can use Refonte Learning’s overview of Python libraries for analytical work.

The main six-row source is intentionally small:

Before inspecting pandas output, the authored expectation is already fixed: x/y must use R1,R2; x/z must use R3,R4; y/z must use R5,R6. Each pair has two rows, but no record is complete on all three features.

A separate positive control contains four complete rows:

Those four records produce the identity correlation matrix from one genuinely common cohort.

The script also carries an independent arithmetic oracle using Python Fraction objects over the authored tuples. For these fixtures, it computes centered cross-products and requires the denominator product to have an exact positive rational square root. This is intentionally narrow: it is not a general arbitrary-precision Pearson implementation.

No external dataset, notebook, R runtime, or unpublished research file is required. The R manual is used later only as statistical documentation about pairwise-complete calculations, not as an executed comparator.

Run the independent twelve-case check

Save the following source exactly as correlation_sample_lab.py. Its SHA-256 is 454b2fe164e236890d58bd319db30bd65cb6464282bbd84af87e9dfcd53ed4d7; it contains 185 lines and 1,001 whitespace-separated words.

python3.12 --version
python3.12 -m venv .venv-corr
.venv-corr/bin/python -m pip install pandas==2.2.3 numpy==2.3.5
.venv-corr/bin/python correlation_sample_lab.py > correlation-sample-evidence.json

The package versions are pinned, while the script records rather than enforces the Python patch release. The supplied reference used Python 3.12.14.

"""Owned Pearson-correlation population controls; no inference or production release."""
import hashlib, json, math, platform, sys
from datetime import datetime, timezone
from fractions import Fraction
from itertools import combinations
from pathlib import Path
import numpy as np
import pandas as pd

COLS = ['x', 'y', 'z']
ROWS = [('R1', -1, -1, None), ('R2', 1, 1, None),
        ('R3', -1, None, -1), ('R4', 1, None, 1),
        ('R5', None, -1, 1), ('R6', None, 1, -1)]
COMPLETE = [('C1', -1, -1, -1), ('C2', -1, 1, 1),
            ('C3', 1, -1, 1), ('C4', 1, 1, -1)]
PAIR_IDS = {'x|y': ['R1', 'R2'], 'x|z': ['R3', 'R4'], 'y|z': ['R5', 'R6']}
EXPECTED = [[1., 1., 1.], [1., 1., -1.], [1., -1., 1.]]
IDENTITY = [[1., 0., 0.], [0., 1., 0.], [0., 0., 1.]]
HELPER_CALLS = 0

def require(ok, message):
    if not ok:
        raise AssertionError(message)

def frame(rows):
    return pd.DataFrame([r[1:] for r in rows], index=[r[0] for r in rows],
                        columns=COLS, dtype='float64')

def matrix(df):
    return [[None if math.isnan(float(x)) else float(x) for x in row]
            for row in df.to_numpy()]

def close_matrix(actual, expected):
    require(len(actual) == len(expected), 'Matrix row count')
    for arow, erow in zip(actual, expected, strict=True):
        require(len(arow) == len(erow), 'Matrix column count')
        for a, e in zip(arow, erow, strict=True):
            require(a is None if e is None else a is not None and abs(a - e) <= 1e-12,
                    'Authored matrix mismatch')

def oracle(rows):
    result = []
    for a in range(1, 4):
        out = []
        for b in range(1, 4):
            pairs = [(Fraction(r[a]), Fraction(r[b])) for r in rows
                     if r[a] is not None and r[b] is not None]
            require(len(pairs) >= 2, 'Oracle requires two complete pairs')
            ma = sum(x for x, in pairs) / len(pairs)
            mb = sum(y for , y in pairs) / len(pairs)
            cross = sum((x - ma) (y - mb) for x, y in pairs)
            product = sum((x - ma)*2 for x, in pairs) sum((y - mb)*2 for , y in pairs)
            rn, rd = math.isqrt(product.numerator), math.isqrt(product.denominator)
            require(rn rn == product.numerator and rd rd == product.denominator and rn > 0,
                    'Oracle is limited to positive rational square-root fixtures')
            out.append(float(cross / Fraction(rn, rd)))
        result.append(out)
    return result

def membership(df):
    return {a + '|' + b: list(df.index[df[[a, b]].notna().all(axis=1)])
            for a, b in combinations(COLS, 2)}

def common_candidate(df):
    global HELPER_CALLS
    if not df.index.is_unique or not all(type(i) is str and i for i in df.index):
        raise ValueError('Invalid record identities')
    if not df.columns.is_unique or not set(COLS).issubset(df.columns):
        raise ValueError('Invalid feature schema')
    selected = df[COLS]
    if any(t != np.dtype('float64') for t in selected.dtypes):
        raise ValueError('Expected float64 features')
    values = selected.to_numpy()
    if np.isinf(values).any() or np.any(np.abs(values) > 10):
        raise ValueError('Unqualified numerical input')
    common = selected.dropna(subset=COLS, how='any')
    ids = list(common.index)
    if len(common) < 3:
        return {'status': 'HOLD_INSUFFICIENT_COMPLETE_ROWS', 'ids': ids, 'matrix': None}
    if any(len(set(common[c])) < 2 for c in COLS):
        return {'status': 'HOLD_ZERO_VARIANCE', 'ids': ids, 'matrix': None}
    HELPER_CALLS += 1
    corr = common.corr(method='pearson', min_periods=3, numeric_only=False)
    require(np.isfinite(corr.to_numpy()).all(), 'Undefined candidate coefficient')
    return {'status': 'ACCEPT_BOUNDED_COHORT_COMPUTATION', 'ids': ids, 'matrix': matrix(corr)}

def main(report):
    require(pd.__version__ == '2.2.3' and np.__version__ == '2.3.5', 'Package baseline')
    df, complete = frame(ROWS), frame(COMPLETE)
    snapshot = df.copy(deep=True)
    cases = report['cases']

    def record(case, observed, expected, meaning):
        require(observed == expected, case + ' mismatch')
        cases.append({'case': case, 'observed': observed, 'expected': expected,
                      'meaning': meaning, 'status': 'PASS'})

    members = membership(df)
    record('C01', members, PAIR_IDS, 'Equal pair counts do not establish equal row identities')
    corr = df.corr(method='pearson', min_periods=2, numeric_only=False)
    close_matrix(matrix(corr), EXPECTED)
    close_matrix(oracle(ROWS), EXPECTED)
    record('C02', bool(np.isfinite(corr.to_numpy()).all()), True, 'All coefficients are finite')
    witness = [-1, 1, 1]
    quadratic = sum(Fraction(float(corr.iat[i, j])) witness[i] witness[j]
                    for i in range(3) for j in range(3))
    record('C03', str(quadratic), '-3', 'This exact matrix is not positive semidefinite')
    floor = df.corr(method='pearson', min_periods=3, numeric_only=False)
    record('C04', matrix(floor), [[1., None, None], [None, 1., None], [None, None, 1.]],
           'A higher pair threshold does not create a common cohort')
    held = common_candidate(df)
    record('C05', held, {'status': 'HOLD_INSUFFICIENT_COMPLETE_ROWS', 'ids': [], 'matrix': None},
           'The source has no row complete on all three features')
    good = common_candidate(complete)
    close_matrix(good['matrix'], oracle(COMPLETE))
    record('C06', good, {'status': 'ACCEPT_BOUNDED_COHORT_COMPUTATION',
           'ids': ['C1', 'C2', 'C3', 'C4'], 'matrix': IDENTITY}, 'A declared complete-cohort control')
    separate = []
    for a, b in combinations(range(3), 2):
        for n, (x, y) in enumerate([(-1, -1), (-1, 1), (1, -1), (1, 1)], 1):
            values = [None, None, None]
            values[a], values[b] = x, y
            separate.append((COLS[a] + COLS[b] + str(n), *values))
    separate_df = frame(separate)
    separate_corr = separate_df.corr(method='pearson', min_periods=3, numeric_only=False)
    close_matrix(matrix(separate_corr), oracle(separate))
    record('C07', [matrix(separate_corr), list(separate_df.dropna(subset=COLS).index)],
           [IDENTITY, []], 'An identity matrix also does not prove a common cohort')
    mixed = frame(ROWS + COMPLETE)
    mixed_pair = mixed.corr(method='pearson', min_periods=3, numeric_only=False)
    close_matrix(matrix(mixed_pair), oracle(ROWS + COMPLETE))
    mixed_common = common_candidate(mixed)
    record('C08', mixed_common, good, 'Explicit common-row selection changes the reported population')
    annotated = complete.assign(note=['ok', 'ok', 'ok', None])
    selected = common_candidate(annotated)
    record('C09', [list(annotated.dropna().index), selected['ids']],
           [['C1', 'C2', 'C3'], ['C1', 'C2', 'C3', 'C4']], 'Unselected metadata must not change cohort membership')
    reordered = df.iloc[::-1][['z', 'x', 'y']]
    rc = reordered.corr(method='pearson', min_periods=2, numeric_only=False)
    record('C10', [list(rc.columns), matrix(rc), membership(reordered)],
           [['z', 'x', 'y'], [[1., 1., -1.], [1., 1., 1.], [-1., 1., 1.]],
            {k: list(reversed(v)) for k, v in PAIR_IDS.items()}], 'Keep feature labels and source identities across reordering')
    duplicate = complete.copy()
    duplicate.index = ['C1', 'C1', 'C3', 'C4']
    infinite = complete.copy()
    infinite.loc['C1', 'x'] = math.inf
    rejected, before = [], HELPER_CALLS
    for label, bad in [('duplicate_ids', duplicate), ('missing_feature', complete.drop(columns='z')),
                       ('infinity', infinite)]:
        try:
            common_candidate(bad)
        except ValueError:
            rejected.append(label)
        else:
            raise AssertionError('Missing admission rejection')
    constant = common_candidate(complete.assign(z=0.))
    record('C11', [rejected, constant['status'], HELPER_CALLS - before],
           [['duplicate_ids', 'missing_feature', 'infinity'], 'HOLD_ZERO_VARIANCE', 0],
           'Input and coefficient eligibility are separate checks before helper correlation')
    with_missing = frame(ROWS + [('R7', None, None, None)])
    ledger = [{'id': i, 'pairs': [k for k, ids in membership(with_missing).items() if i in ids],
               'complete': bool(with_missing.loc[i, COLS].notna().all())} for i in with_missing.index]
    record('C12', [membership(with_missing), ledger[-1], len(ledger)],
           [PAIR_IDS, {'id': 'R7', 'pairs': [], 'complete': False}, 7], 'Retain records contributing to no coefficient')
    require(df.equals(snapshot), 'Original source changed')
    require(HELPER_CALLS == 3, 'Helper correlation-call inventory')
    require([c['case'] for c in cases] == [f'C{i:02d}' for i in range(1, 13)], 'Case completeness')
    report.update(status='PASS_12_CASES', source_rows=ROWS, complete_rows=COMPLETE,
                  main_matrix=matrix(corr), main_membership=members,
                  mixed_pairwise_matrix=matrix(mixed_pair), mixed_common=mixed_common,
                  separate_pair_membership=membership(separate_df), row_ledger=ledger,
                  helper_correlation_calls=HELPER_CALLS, source_preserved=True)

if name == '__main__':
    report = {'status': 'FAIL', 'cases': [], 'started_utc': datetime.now(timezone.utc).isoformat(),
              'source_sha256': hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),
              'environment': {'python': sys.version, 'pandas': pd.__version__,
                              'numpy': np.__version__, 'platform': platform.platform()}}
    try:
        main(report)
    except Exception as error:
        report['error'] = {'type': type(error).__name__, 'message': str(error)}
    report['finished_utc'] = datetime.now(timezone.utc).isoformat()
    print(json.dumps(report, indent=2, allow_nan=False))
    sys.exit(0 if report['status'] == 'PASS_12_CASES' else 1)

The supplied reference run started at 2026-10-31 07:14:09.039229 UTC, finished at 07:14:09.082865 UTC, exited with status 0, and recorded PASS_12_CASES. Those timestamps describe one reference execution, not a speed benchmark. The source hash identifies the bytes that were run; by itself it does not authenticate when or where an execution occurred.

Preserve the execution record

Keep the generated correlation-sample-evidence.json with the exit status. The script begins in FAIL, verifies pandas 2.2.3 and NumPy 2.3.5, records the interpreter and platform, runs all twelve controls, and returns zero only after reaching PASS_12_CASES.

The JSON should retain observed and expected values rather than only a final green status. That makes a future disagreement investigable: a reviewer can tell whether membership, arithmetic, schema admission, or another control changed.

The reference environment is the supplied Python 3.12.14/glibc 2.39 execution. A separate check of the identical hashed source used Python 3.13.5 with pandas 2.2.3 and NumPy 2.3.5 and also returned PASS_12_CASES; that later check is not the declared reference baseline and does not expand the tested environment claim.

The script provides bounded computational evidence only. ACCEPT_BOUNDED_COHORT_COMPUTATION means the authored helper admitted the fixture and returned its candidate. It does not authorize a scientific conclusion, establish sampling quality, or approve publication.

Attach an identity ledger to every coefficient

C01 answers the article’s central question directly.

Those memberships come from applying .notna().all(axis=1) to each selected two-column slice. pandas documents notna() as returning a Boolean mask for existing, non-missing values; None and NaN are treated as missing under the documented behavior.

The ledger is more informative than a bare n=2 because it preserves the coefficient’s feature labels and its source identities together. In this fixture, corr(x,y)=1, corr(x,z)=1, and corr(y,z)=-1, but those values do not come from one six-row calculation or one common two-row subset. They come from three different two-record intersections.

The expected pair sets were declared independently in PAIR_IDS, before the computed membership result was accepted. The Fraction oracle independently checks the authored scalar arithmetic on the original tuples. Agreement between the pandas result and the oracle supports the calculation for these controls; neither mechanism supplies analytical approval.

Equal counts can conceal different records

Every off-diagonal coefficient has exactly two contributing records. A correlation matrix sample-count table would therefore show three matching counts and still miss the essential fact: no off-diagonal pair shares an ID with another.

This is why count equality, matrix symmetry, and coefficient finiteness answer different questions. Counts test cardinality. Labels preserve semantic position. IDs establish membership. Finiteness shows that a numeric result exists. None of the latter three, individually or together, prove that all coefficients used one common set of source rows.

Follow min_periods through a deliberate failing case

C02 computes the main Pearson table with min_periods=2:

All nine entries are finite, and the authored literal expectation agrees with the independent Fraction calculation. That is useful arithmetic evidence, but it does not change C01’s membership finding.

C04 then raises min_periods to 3. pandas defines that argument as the minimum observations required per pair. Each feature individually has four observed values, so this fixture retains a unit diagonal. Each distinct off-diagonal pair has only two jointly observed records, so all three off-diagonal coefficients become missing. The JSON serializer converts those missing coefficients to null.

The important boundary is that the higher threshold did not create any complete records. It only made the three off-diagonal pairs ineligible.

C05 applies the authored common-cohort policy to the same six-row source. dropna(subset=COLS, how='any') finds zero rows observed on all of x, y, and z, so the candidate is:

HOLD_INSUFFICIENT_COMPLETE_ROWS

That is the correct computational response for the declared common-cohort policy in this fixture: no estimate is fabricated.

The diagonal result in C04 should not be generalized into a claim that every degenerate Pearson column always returns one. The pandas documentation includes diagonal wording in the broader corr() API, including custom callable behavior, but this article does not transfer that wording into a guarantee about zero-variance Pearson cases.

Check what a plausible matrix fails to establish

C03 examines the exact main matrix rather than judging it by appearance:

For the authored witness , exact Fraction arithmetic gives:

A positive-semidefinite matrix cannot have a negative quadratic form, so this literal matrix is not positive semidefinite. The witness proves only that fact about this exact fixture; it is not presented as a general PSD-testing algorithm.

The broader phenomenon is documented outside this laboratory. The R stats manual explains that pairwise.complete.obs computes each pair from its available complete observations and warns that the resulting covariance or correlation matrix can be non-positive-semidefinite. The consulted R-devel page identifies stats version 4.6.0 in its footer. No R process was executed here, and R’s options are not being attributed to pandas.

pandas’ 2.2.3 covariance documentation similarly notes that pairwise missing-value exclusion can produce a covariance estimate that is not guaranteed to be positive semidefinite. That is contextual documentation about DataFrame.cov(), not a covariance workflow executed by this article.

An identity matrix can still have no complete row

C07 supplies the more important counterexample. Twelve records are created in three four-row groups. One group contains only x/y, another only x/z, and another only y/z. Within each pair, the four balanced combinations (-1,-1), (-1,1), (1,-1), and (1,1) yield zero Pearson correlation.

The resulting matrix is exactly the identity:

Yet the complete-row set is empty because every source record is missing the third feature.

C06 produces that same identity matrix from a different construction: four records, C1–C4, all complete on all three features. Its common cohort is therefore explicitly ['C1','C2','C3','C4'].

The comparison is decisive for sample auditing. One identity matrix comes from a genuine shared four-row sample; another comes from twelve pair-only records with no complete row at all. Positive definiteness, therefore, cannot establish a common cohort. Algebraic appearance is not a substitute for row-membership evidence.

Choose an explicit common cohort when the contract requires one

C08 combines R1–R6 with complete rows C1–C4. On that mixed source, pandas pairwise Pearson correlation uses all rows eligible for each pair. The off-diagonals are approximately +1/3, +1/3, and -1/3.

The authored common-cohort helper makes a different selection: it keeps only C1–C4 because those are the records observed on all three required features. Their separately computed matrix is the identity, with zero off-diagonals.

These are not competing implementations of one identical sample. They are coefficients computed from different observed record sets. The pairwise version includes the eligible R records alongside C1–C4 for each pair; the common-cohort candidate uses C1–C4 only.

Mechanically, DataFrame.dropna() can drop rows on axis 0 when missingness is present, and its subset argument limits which columns are considered. Using subset=['x','y','z'] therefore makes the feature contract explicit. That operation selects a different observed sample; pandas does not claim that the selected rows are representative, sufficiently numerous, or free of selection bias.

A common cohort is consequently not a universal correction for pairwise analysis. If the analysis owner declared pairwise descriptive reporting, replacing it silently with complete cases changes the estimand’s observed sample definition. If the owner requires a common sample but the complete set is too small or otherwise unqualified, the responsible result is a hold, not an invented substitute.

This is consistent with the broader principle of preserving source meaning during data cleaning: transformations should follow declared field requirements rather than erase records merely because some unrelated field is missing.

Keep optional metadata outside the feature contract

C09 makes that point concrete. The four complete feature rows receive an extra note column, with C4’s note missing.

A broad annotated.dropna() would retain only C1–C3. The authored helper instead selects x, y, and z first and calls dropna(subset=COLS, how='any'); it retains C1–C4. pandas documents precisely that subset determines which labels are considered for row dropping.

Under this contract, note is not an analytical feature and therefore must not silently redefine membership. That does not mean optional metadata should always be ignored. It means the columns that govern cohort admission must be named explicitly.

Reject unqualified input before computing a candidate

A complete-row selector is only one stage of the helper. The code deliberately separates input admission from coefficient eligibility.

First comes identity and schema admission. Record IDs must be unique, nonempty strings; the required columns must exist; selected feature labels must be unique; and the selected dtypes must be float64. These are authored teaching policies, not pandas requirements.

Second comes numerical admission. The helper rejects infinity and values whose absolute magnitude exceeds 10. DataFrame.notna() cannot replace that check: pandas 2.2.3 explicitly documents that numpy.inf and empty strings are ordinarily considered non-missing unless the infinity-related option is changed. None and NaN, by contrast, are missing.

Third comes cohort eligibility. The helper selects rows complete on x/y/z; fewer than three produces HOLD_INSUFFICIENT_COMPLETE_ROWS. Again, three is only a fixture policy. It does not establish adequate power, representativeness, or scientific sufficiency.

Fourth comes variance eligibility. If any selected feature has fewer than two distinct values in the common cohort, the candidate is held as HOLD_ZERO_VARIANCE. That prevents the helper from treating a constant-column Pearson coefficient as usable merely because missingness and row count passed.

Finally, after corr() is called, the helper requires every returned candidate coefficient to be finite.

C11 exercises several of those boundaries. Duplicate IDs, a missing z feature, and an infinite input are rejected before helper correlation. A complete frame with constant z reaches the eligibility stage but returns HOLD_ZERO_VARIANCE. The helper-call counter does not increase during those C11 controls.

Exactly three calls to corr() occur inside common_candidate across the full reference script: C06, C08, and C09. Direct baseline and comparison calculations elsewhere in main() are not counted as helper calls.

The arithmetic oracle has limits too. It works directly from the authored tuples, requires positive rational square roots for its fixtures, and compares the resulting small matrices with a local 1e-12 allowance. It is not a general high-precision correlation engine.

Guard coverage is also intentionally incomplete. The executed negative controls include duplicate IDs, a missing required feature, infinity, and a constant feature. Duplicate column labels, every malformed-ID form, a wrong selected dtype, and the magnitude-limit rejection were not separately injected as recorded cases. The frame() constructor creates float64 values; it does not validate original lexical input from a file or external system.

Reconcile every source record and preserve labels

C10 tests both dimensions of identity. The original six rows are reversed, while the feature order is changed to z, x, y. The resulting correlation table follows that feature order, and the pair-membership evidence follows the reversed row order. Because the evidence carries feature names and record IDs, neither transformation creates positional ambiguity.

That matters in handovers. A naked numeric array can be accidentally interpreted under the wrong column order; a set of counts can be detached from the IDs it summarizes. The audit artifact should keep both labels attached.

C12 tests source preservation in another direction. It adds R7, whose x, y, and z are all missing. The pair memberships remain:

• x|y: R1, R2

• x|z: R3, R4

• y|z: R5, R6

But the row ledger contains seven entries, including:

{'id': 'R7', 'pairs': [], 'complete': False}

That distinction is essential. source_rows in the report refers to the original six-row baseline. row_ledger refers to the seven-row C12 variant. The script does not blur those scopes.

This preservation habit resembles the concern in Refonte Learning’s article on row reconciliation for missing grouping keys, but correlation membership is structurally different from a grouping partition. A grouped row ordinarily belongs to one key combination under a given grouping policy; a correlation-source record may contribute to zero, one, or several feature pairs.

Consequently, summing pair counts is not a source-row reconciliation equation. The correct evidence is a row ledger showing each source ID and the pair memberships to which it contributes.

Rebuild a disputed report from preserved evidence

When an existing correlation report is disputed, do not begin by overwriting missing values or substituting a new complete-case matrix under the old label. Rebuild from preserved evidence.

A practical recovery sequence is:

1. Preserve the original DataFrame or its governed source snapshot and retain the previously reported candidate.

2. Record the intended features, method, min_periods, ID semantics, and sample policy.

3. Reconstruct pair memberships from the preserved source.

4. If a common cohort is required, derive it separately with the declared feature subset.

5. Produce a separately labelled candidate rather than overwriting the old artifact.

6. Compare coefficients, pair IDs, complete IDs, and any records contributing to no estimate.

7. Route any sample-policy change to the appropriate data, statistical, or domain owner before replacing an approved report.

The difference between the mixed pairwise matrix and the C1–C4 common candidate demonstrates why labelling matters. Approximately ±1/3 does not become zero because one calculation is “more correct”; zero appears because the observed-record selection changed.

That provenance discipline is closely related to distinguishing introduced and missing report values: a changed output should remain traceable to the operation and policy that changed it, rather than being presented as if it came unchanged from the source.

The laboratory’s preservation assertion is narrower than a production recovery policy. It verifies that the original six-row df still equals its deep-copied snapshot after the scripted controls. It does not establish immutability for every constructed variant, a versioned publication system, rollback procedure, concurrency behavior, or disaster recovery process.

If the replacement calculation is held, preserve the earlier approved artifact as historical evidence rather than silently substituting an unqualified result.

Use a decision table for pairwise, common-cohort and held results

The final action should follow the declared contract and the recorded evidence, not a matrix property chosen after seeing the answer.

ACCEPT_BOUNDED_COHORT_COMPUTATION is deliberately narrow. It says the teaching helper admitted the input and computed its bounded candidate. It does not mean “approved for inference,” “representative,” or “ready to publish.”

Likewise, HOLD is a real outcome. Replacing a hold with an arbitrary coefficient, zero, copied value, or broadened sample would destroy the evidence the review is supposed to preserve.

Assign the decision to the right owner

The analyst owns the computation record: versions, source IDs, memberships, code, expected-versus-observed controls, and candidate labels.

The data owner clarifies what the record identifiers and fields mean and whether a row belongs to the intended analytical source. The statistical or domain reviewer decides whether pairwise estimates or a common observed cohort answer the intended question, and whether the available sample is adequate for that use.

If the sample policy changes, the person or governance process authorized to change the analytical specification must approve it. min_periods, a positive-definite matrix, or a convenient complete-row count cannot supply that authority by themselves.

Hand over a reproducible analytical record

A useful correlation handover should allow another reviewer to answer both “what was computed?” and “from which records?” without reconstructing assumptions from a spreadsheet.

The acceptance packet should include the source identity or governed snapshot reference; the exact selected feature names and their order; the Pearson method and min_periods; the declared sample policy; pair IDs and counts; common-cohort IDs when relevant; records belonging to no pair; missing-value and constant-column decisions; expected-versus-observed control results; interpreter, pandas and NumPy versions; platform; source hash; execution status; and the owner who approved the sample policy.

The twelve recorded controls can be summarized compactly:

A future pandas, NumPy, Python, input-schema, identity-policy, or cohort-rule change should trigger revalidation of the controls affected by that change. This laboratory tested pandas 2.2.3 and NumPy 2.3.5 under the stated reference environment; it does not claim equivalent behavior was qualified for other releases.

Arithmetic agreement and analytical approval should remain separate in the handover. The Fraction oracle can tell a reviewer that the authored fixtures agree with the calculated coefficients. It cannot establish whether the records represent the intended subject matter, whether missingness is acceptable, or whether a scientific claim should be published.

Build the statistical programming skills behind this review

This kind of review combines ordinary Python/Pandas mechanics with stronger analytical habits: explicit data contracts, stable identities, controlled missingness selection, independent expectations, and a documented decision boundary between “computable” and “approved.”

Refonte Learning’s current Data Science & AI programme lists a three-month format at 12–14 hours per week and names Python Data Science, Statistical Modelling, Model Optimization & Problem-Solving, EDA and Data Visualization, Machine Learning/Predictive Modelling, and application to industry projects among its competencies. Its FAQ names Python, Jupyter Notebook, Pandas, NumPy, Matplotlib, scikit-learn, and TensorFlow, and separately mentions R; its visible educational preview includes introductory statistical concepts and descriptive statistics in Excel.

Those page-level facts should not be stretched into claims about this specialist workflow. They do not confirm whether pandas corr, pair-membership auditing, complete-case policy, PSD counterexamples, Fraction, or this exact laboratory is taught. Current package versions, the depth of missing-data review, admission terms, duration wording beyond the main three-month listing, mentor cadence, internship conditions, external placement, and employment outcomes remain unconfirmed; the degree-oriented admission requirement and FAQ wording differ. The page names Dr. John Anderson in its Educational Mentors section and lists a Training Certificate and Certificate of Internship on successful completion, but those statements do not independently establish supervision cadence or placement conditions.

For this review, the sensible next action is smaller and immediately testable: attach the pair-ID ledger to the correlation table, write down whether the contract permits pairwise samples or requires one common cohort, and hold any candidate whose evidence does not satisfy that declaration.