StandardScaler Partial Fits Changed the Variance. Did the Weights Match?

A successful StandardScaler.partial_fit() call is a weak acceptance signal. The method can return normally, update mean_, var_, and n_samples_seen_, and leave a numerically plausible scaler even when one batch was supplied with the wrong weighting policy. That is the operational failure boundary tested here: weighted incremental mean and variance across StandardScaler.partial_fit batches, not StandardScaler generally, not downstream model quality, and not preprocessing architecture.

The documented contract matters. In scikit-learn 1.8.0, partial_fit(X, sample_weight=...) supports per-sample weights; n_samples_seen_ accumulates the weight sum when weights are supplied; NaNs are disregarded during fitting; and var_ uses the population-style convention corresponding to ddof=0. The API documentation also identifies Chan, Golub and LeVeque’s incremental variance work as the basis for the online calculation. The version-linked implementation source reached from the 1.8.0 documentation resolves to commit fe2edb3cdbd75ae4e662fda67dcb19277258792b, which is materially stronger evidence for this run than an unpinned main branch.

This playbook therefore treats API success as necessary but insufficient. The acceptance boundary is an independently authored rational-arithmetic manifest, a synthetic disposable fixture, explicit batch and weight identities, observed estimator state, warning and exception capture, and a case-by-case evidence ledger. I executed that fixture locally for this article on CPython 3.13.5, scikit-learn 1.8.0 and NumPy 2.3.5. Where this article reports an observation, it comes from that recorded run; documentary claims remain separate from those observations.

Define the acceptance contract and pin the reproducible baseline

The contract is deliberately narrower than “StandardScaler works.” The release question is: given an ordered set of batches and a declared weight vector for every batch, does incremental fitting produce the independently expected weighted mean, weighted population variance and accumulated weight count for every feature?

For samples  with declared nonnegative test weights , this fixture’s independent oracle uses:

and

That denominator is intentional. StandardScaler documents its variance convention as equivalent to numpy.std(..., ddof=0), so a sample-variance oracle using n = 1 would be the wrong acceptance reference. The oracle below is not NumPy’s average, var, or any scikit-learn helper. It uses Python Fraction arithmetic over independently authored expected values, then converts the final rational numbers only for comparison.

The accepted input inventory is two features and four canonical rows:

The canonical batching is B1 = {r1,r2} and B2 = {r3,r4}. Independently, the expected total weight is 10, weighted means are 28/5 = 5.6 and 96/5 = 19.2, and weighted variances are 221/25 = 8.84 and 884/25 = 35.36.

The assertions are not merely “no exception” and “finite numbers.” Acceptance requires all three estimator-state dimensions (mean_, var_, and n_samples_seen_) to match the manifest within rtol=1e-12, atol=1e-12. The documentation specifically says n_samples_seen_ becomes a floating weight sum when sample weights are used and can become feature-specific when missing values occur. That makes the count state part of the evidence, not incidental metadata.

The disposable execution baseline was:

This distinction between a versioned runtime and the moving stable documentation matters. As of this research, scikit-learn’s stable StandardScaler page identifies itself as 1.9.1, whereas the executed fixture is 1.8.0. The relevant public contract is consistent across those pages (partial_fit accepts sample_weight, n_samples_seen_ accumulates weights, and the variance convention remains ddof=0), but a release record should cite the documentation matching the tested runtime, not silently substitute a later “stable” selector.

The function name sklearn.utils.extmath.incremental_mean_and_var requires clarification. The current API index does not expose a public function under that exact name; current source references the private helper incrementalmean_and_var, while sparse incremental statistics have the public sklearn.utils.sparsefuncs.incr_mean_variance_axis API. Treat that as a version/API-surface clarification, not as evidence against StandardScaler’s documented weighted behavior.

For a negative control, the harness deliberately computes an unweighted population mean and variance from the same four rows. That false oracle gives means [4.5, 17.0] and variances [8.75, 35.0], which the weighted acceptance manifest must reject. In the executed fixture, that negative control was rejected as intended. An acceptance harness that cannot distinguish those values from the weighted target has no standing to approve a weighted migration.

This contract is intentionally different from Refonte Learning’s scikit-learn OneHotEncoder unknown categories investigation. That article preserves raw categorical identity to detect a representational collision between a dropped baseline and an unseen category; its recorded fixture used scikit-learn 1.8.0 but tests encoding identity, warning behavior and inverse ambiguity, not weighted numeric accumulation. Reusing its pass/fail mechanism here would answer the wrong question.

Author the expected-case manifest before exercising the scaler

The expected-case manifest is the control plane of the test. Write it before looking at StandardScaler.mean_ or var_; otherwise the implementation under test can quietly become its own oracle.

Seven cases are sufficient for the bounded release question:

The dangerous case is omitted_second_batch_weights. Batch one is supplied with [1,3]; batch two is deliberately sent without sample_weight. StandardScaler documents sample_weight as optional, so omission is not inherently an API error. Operationally, though, the intended contract says every row has the declared weights [1,3,2,4]. Omitting weights from the second batch therefore changes the statistical problem without necessarily changing control flow.

That distinction is exactly why a one-shot comparison to StandardScaler.fit() is not the primary oracle here. In scikit-learn’s implementation, fit resets internal state and delegates through the same estimator machinery used for fitting statistics; using one StandardScaler pathway to certify another would provide useful corroboration but insufficient independence. The version-linked source is the appropriate place to inspect implementation flow, while the expected values remain independently authored.

The manifest self-check recomputes those literals with Python Fraction. If the independently written fractions disagree with the recomputation, execution stops before invoking scikit-learn. That catches transcription errors in the manifest itself.

A similar separation of “expected inventory” from “observed state” appears in the Refonte SQLite foreign key enforcement article, but the evidence workflow is not reusable wholesale. That SQLite case proves whether a connection setting actually took effect relative to transaction state and uses valid-versus-orphan writes as behavioral probes. Here there is no connection-admission state machine: the identity that must survive is row-to-weight association across incremental batches.

The complete independently authored manifest section of the fixture is:

EXPECTED = {
    "correct_two_batch": {
        "mean": ["28/5", "96/5"],
        "var": ["221/25", "884/25"],
        "n_samples_seen": ["10", "10"],
        "decision": "ACCEPT",
    },
    "omitted_second_batch_weights": {
        "mean": ["4", "16"],
        "var": ["19/3", "76/3"],
        "n_samples_seen": ["6", "6"],
        "decision": "REPAIR",
    },
    "partition_invariance": {
        "mean": ["28/5", "96/5"],
        "var": ["221/25", "884/25"],
        "n_samples_seen": ["10", "10"],
        "decision": "ACCEPT",
    },
    "scaled_weights_x10": {
        "mean": ["28/5", "96/5"],
        "var": ["221/25", "884/25"],
        "n_samples_seen": ["100", "100"],
        "decision": "ACCEPT",
    },
    "zero_weight_outlier": {
        "mean": ["28/5", "96/5"],
        "var": ["221/25", "884/25"],
        "n_samples_seen": ["10", "10"],
        "decision": "ACCEPT",
    },
    "nan_feature_specific": {
        "mean": ["28/5", "150/7"],
        "var": ["221/25", "1664/49"],
        "n_samples_seen": ["10", "7"],
        "decision": "ACCEPT",
    },
    "bad_weight_length": {
        "exception_type": "ValueError",
        "decision": "ACCEPT",
    },
}

The exact-rational oracle deliberately represents missing feature values as None rather than asking NumPy or scikit-learn to decide how to handle them:

from fractions import Fraction

def frac_weighted_stats(rows, weights):
    n_features = len(rows[0])
    means, variances, weight_sums = [], [], []

    for j in range(n_features):
        pairs = []
        for row, weight in zip(rows, weights):
            value = row[j]
            if value is None:
                continue
            pairs.append(
                (Fraction(str(value)), Fraction(str(weight)))
            )

        total_weight = sum(
            (weight for , weight in pairs), Fraction(0)
        )
        mean = sum(
            (weight value for value, weight in pairs),
            Fraction(0),
        ) / totalweight
        variance = sum(
            (
                weight (value - mean) ** 2
                for value, weight in pairs
            ),
            Fraction(0),
        ) / total_weight

        means.append(mean)
        variances.append(variance)
        weight_sums.append(total_weight)

    return means, variances, weight_sums

The missing-value boundary is grounded in the estimator documentation: StandardScaler says NaNs are disregarded during fit, and n_samples_seen_ can therefore be feature-specific. This case is not an imputation test; it exists only to ensure the evidence ledger does not wrongly force one scalar count when weighted feature coverage differs.

Construct the fixture and reproduce the misleading success

The full fixture below is executable as a single file. It uses synthetic arrays only, writes no production state, captures warnings, checks malformed weight cardinality, records environment metadata and hashes its own source.

from future import annotations

import hashlib
import json
import platform
import warnings
from fractions import Fraction
from pathlib import Path

import numpy as np
import sklearn
from sklearn.preprocessing import StandardScaler

ATOL = 1e-12
RTOL = 1e-12
FIXTURE_REVISION = "standardscaler-weighted-partial-fit-v1"

EXPECTED = {
    "correct_two_batch": {
        "mean": ["28/5", "96/5"],
        "var": ["221/25", "884/25"],
        "n_samples_seen": ["10", "10"],
        "decision": "ACCEPT",
    },
    "omitted_second_batch_weights": {
        "mean": ["4", "16"],
        "var": ["19/3", "76/3"],
        "n_samples_seen": ["6", "6"],
        "decision": "REPAIR",
    },
    "partition_invariance": {
        "mean": ["28/5", "96/5"],
        "var": ["221/25", "884/25"],
        "n_samples_seen": ["10", "10"],
        "decision": "ACCEPT",
    },
    "scaled_weights_x10": {
        "mean": ["28/5", "96/5"],
        "var": ["221/25", "884/25"],
        "n_samples_seen": ["100", "100"],
        "decision": "ACCEPT",
    },
    "zero_weight_outlier": {
        "mean": ["28/5", "96/5"],
        "var": ["221/25", "884/25"],
        "n_samples_seen": ["10", "10"],
        "decision": "ACCEPT",
    },
    "nan_feature_specific": {
        "mean": ["28/5", "150/7"],
        "var": ["221/25", "1664/49"],
        "n_samples_seen": ["10", "7"],
        "decision": "ACCEPT",
    },
    "bad_weight_length": {
        "exception_type": "ValueError",
        "decision": "ACCEPT",
    },
}

B1 = np.array([[1.0, 10.0], [3.0, 14.0]])
W1 = np.array([1.0, 3.0])
B2 = np.array([[5.0, 18.0], [9.0, 26.0]])
W2 = np.array([2.0, 4.0])


def parse_fracs(values):
    return np.array(
        [float(Fraction(value)) for value in values],
        dtype=float,
    )


def frac_weighted_stats(rows, weights):
    """Independent exact-rational weighted oracle."""
    n_features = len(rows[0])
    means, variances, sums = [], [], []

    for feature in range(n_features):
        pairs = []
        for row, weight in zip(rows, weights):
            value = row[feature]
            if value is None:
                continue
            pairs.append(
                (Fraction(str(value)), Fraction(str(weight)))
            )

        total_weight = sum(
            (weight for , weight in pairs), Fraction(0)
        )
        mean = sum(
            (weight value for value, weight in pairs),
            Fraction(0),
        ) / totalweight

        variance = sum(
            (
                weight (value - mean) ** 2
                for value, weight in pairs
            ),
            Fraction(0),
        ) / total_weight

        means.append(mean)
        variances.append(variance)
        sums.append(total_weight)

    return means, variances, sums


def assert_manifest_self_consistency():
    rows = [[1, 10], [3, 14], [5, 18], [9, 26]]
    weights = [1, 3, 2, 4]

    mean, var, count = frac_weighted_stats(rows, weights)
    assert [str(x) for x in mean] == \
        EXPECTED["correct_two_batch"]["mean"]
    assert [str(x) for x in var] == \
        EXPECTED["correct_two_batch"]["var"]
    assert [str(x) for x in count] == \
        EXPECTED["correct_two_batch"]["n_samples_seen"]

    mixed_weights = [1, 3, 1, 1]
    mean, var, count = frac_weighted_stats(
        rows, mixed_weights
    )
    assert [str(x) for x in mean] == \
        EXPECTED["omitted_second_batch_weights"]["mean"]
    assert [str(x) for x in var] == \
        EXPECTED["omitted_second_batch_weights"]["var"]
    assert [str(x) for x in count] == \
        EXPECTED["omitted_second_batch_weights"]["n_samples_seen"]

    nan_rows = [
        [1, 10],
        [3, None],
        [5, 18],
        [9, 26],
    ]
    mean, var, count = frac_weighted_stats(
        nan_rows, weights
    )
    assert [str(x) for x in mean] == \
        EXPECTED["nan_feature_specific"]["mean"]
    assert [str(x) for x in var] == \
        EXPECTED["nan_feature_specific"]["var"]
    assert [str(x) for x in count] == \
        EXPECTED["nan_feature_specific"]["n_samples_seen"]


def state(scaler):
    n = np.asarray(
        scaler.n_samples_seen_,
        dtype=float,
    )
    if n.ndim == 0:
        n = np.repeat(n, len(scaler.mean_))

    return {
        "mean": np.asarray(
            scaler.mean_, dtype=float
        ).tolist(),
        "var": np.asarray(
            scaler.var_, dtype=float
        ).tolist(),
        "n_samples_seen": n.tolist(),
    }


def check_state(name, scaler):
    observed = state(scaler)
    expected = EXPECTED[name]

    for field in (
        "mean",
        "var",
        "n_samples_seen",
    ):
        want = parse_fracs(expected[field])
        got = np.asarray(
            observed[field],
            dtype=float,
        )

        if not np.allclose(
            got,
            want,
            rtol=RTOL,
            atol=ATOL,
            equal_nan=False,
        ):
            raise AssertionError(
                f"{name} {field}: "
                f"got {got}, expected {want}"
            )

    return observed


def run_case(name, fn):
    record = {
        "case": name,
        "status": "OBSERVED",
        "warnings": [],
    }

    try:
        with warnings.catch_warnings(record=True) as caught:
            warnings.simplefilter("always")
            payload = fn()

        record.update(payload)
        record["warnings"] = [
            {
                "category": type(w.message).__name__,
                "message": str(w.message),
            }
            for w in caught
        ]
        record["assertions"] = "PASS"

    except Exception as exc:
        record["assertions"] = "FAIL"
        record["exception"] = {
            "type": type(exc).__name__,
            "message": str(exc),
        }

    return record


def case_correct_two_batch():
    scaler = StandardScaler()

    scaler.partial_fit(
        B1,
        sample_weight=W1,
    )
    before_second = state(scaler)

    scaler.partial_fit(
        B2,
        sample_weight=W2,
    )

    return {
        "before_second_batch": before_second,
        "after": check_state(
            "correct_two_batch", scaler
        ),
    }


def case_omitted_second_batch_weights():
    scaler = StandardScaler()

    scaler.partial_fit(
        B1,
        sample_weight=W1,
    )

    # Deliberate workflow defect.
    scaler.partial_fit(B2)

    after = check_state(
        "omitted_second_batch_weights",
        scaler,
    )

    intended_mean = parse_fracs(
        EXPECTED["correct_two_batch"]["mean"]
    )
    intended_var = parse_fracs(
        EXPECTED["correct_two_batch"]["var"]
    )

    assert not np.allclose(
        after["mean"],
        intended_mean,
        rtol=RTOL,
        atol=ATOL,
    )
    assert not np.allclose(
        after["var"],
        intended_var,
        rtol=RTOL,
        atol=ATOL,
    )

    return {
        "api_returned_without_exception": True,
        "after": after,
        "matches_intended_weighted_contract": False,
    }


def case_partition_invariance():
    scaler = StandardScaler()

    batches = [
        (B1[:1], W1[:1]),
        (
            np.vstack([B1[1:], B2[:1]]),
            np.hstack([W1[1:], W2[:1]]),
        ),
        (B2[1:], W2[1:]),
    ]

    for X, weights in batches:
        scaler.partial_fit(
            X,
            sample_weight=weights,
        )

    return {
        "after": check_state(
            "partition_invariance",
            scaler,
        )
    }


def case_scaled_weights_x10():
    scaler = StandardScaler()

    scaler.partial_fit(
        B1,
        sample_weight=W1 10,
    )
    scaler.partial_fit(
        B2,
        sample_weight=W2 10,
    )

    return {
        "after": check_state(
            "scaled_weights_x10",
            scaler,
        )
    }


def case_zero_weight_outlier():
    scaler = StandardScaler()

    scaler.partial_fit(
        B1,
        sample_weight=W1,
    )

    X2 = np.vstack([
        B2,
        [[1_000_000.0, -1_000_000.0]],
    ])
    weights2 = np.hstack([W2, [0.0]])

    scaler.partial_fit(
        X2,
        sample_weight=weights2,
    )

    return {
        "after": check_state(
            "zero_weight_outlier",
            scaler,
        )
    }


def case_nan_feature_specific():
    scaler = StandardScaler()

    batch1 = np.array([
        [1.0, 10.0],
        [3.0, np.nan],
    ])

    scaler.partial_fit(
        batch1,
        sample_weight=W1,
    )
    scaler.partial_fit(
        B2,
        sample_weight=W2,
    )

    return {
        "after": check_state(
            "nan_feature_specific",
            scaler,
        )
    }


def case_bad_weight_length():
    scaler = StandardScaler()

    try:
        scaler.partial_fit(
            B1,
            sample_weight=np.array([1.0]),
        )
    except Exception as exc:
        assert type(exc).__name__ == \
            EXPECTED["bad_weight_length"][
                "exception_type"
            ]

        return {
            "caught_expected_exception": {
                "type": type(exc).__name__,
                "message": str(exc),
            }
        }

    raise AssertionError(
        "Expected bad sample_weight length to raise"
    )


def negative_control():
    X = np.vstack([B1, B2])

    wrong_mean = X.mean(axis=0)
    wrong_var = X.var(
        axis=0,
        ddof=0,
    )

    correct_mean = parse_fracs(
        EXPECTED["correct_two_batch"]["mean"]
    )
    correct_var = parse_fracs(
        EXPECTED["correct_two_batch"]["var"]
    )

    rejected = not (
        np.allclose(
            wrong_mean,
            correct_mean,
            rtol=RTOL,
            atol=ATOL,
        )
        and np.allclose(
            wrong_var,
            correct_var,
            rtol=RTOL,
            atol=ATOL,
        )
    )

    assert rejected

    return {
        "wrong_unweighted_mean":
            wrong_mean.tolist(),
        "wrong_unweighted_var":
            wrong_var.tolist(),
        "negative_control_rejected": True,
    }


def main():
    assert_manifest_self_consistency()

    source_path = Path(__file__)
    source_sha256 = hashlib.sha256(
        source_path.read_bytes()
    ).hexdigest()

    report = {
        "fixture_revision": FIXTURE_REVISION,
        "fixture_sha256": source_sha256,
        "environment": {
            "python":
                platform.python_version(),
            "python_implementation":
                platform.python_implementation(),
            "platform":
                platform.platform(),
            "numpy":
                np.__version__,
            "scikit_learn":
                sklearn.__version__,
        },
        "tolerance": {
            "rtol": RTOL,
            "atol": ATOL,
        },
        "manifest": EXPECTED,
        "negative_control": run_case(
            "negative_control",
            negative_control,
        ),
        "cases": [
            run_case(
                "correct_two_batch",
                case_correct_two_batch,
            ),
            run_case(
                "omitted_second_batch_weights",
                case_omitted_second_batch_weights,
            ),
            run_case(
                "partition_invariance",
                case_partition_invariance,
            ),
            run_case(
                "scaled_weights_x10",
                case_scaled_weights_x10,
            ),
            run_case(
                "zero_weight_outlier",
                case_zero_weight_outlier,
            ),
            run_case(
                "nan_feature_specific",
                case_nan_feature_specific,
            ),
            run_case(
                "bad_weight_length",
                case_bad_weight_length,
            ),
        ],
    }

    report["overall"] = (
        "PASS"
        if all(
            item["assertions"] == "PASS"
            for item in [
                report["negative_control"],
                *report["cases"],
            ]
        )
        else "FAIL"
    )

    print(
        json.dumps(
            report,
            indent=2,
            sort_keys=True,
            allow_nan=False,
        )
    )

    if report["overall"] != "PASS":
        raise SystemExit(1)


if name == "__main__":
    main()

The misleading-success reproduction is the line that matters:

scaler.partial_fit(B1, sample_weight=W1)
scaler.partial_fit(B2)  # weights accidentally omitted

In the executed run, those two calls returned without an exception or warning. The resulting observed state was:

mean_           = [4.0, 16.0]
var_            = [6.333333333333333,
                   25.333333333333332]
n_samples_seen_ = 6.0

The exact independently expected equivalents for what was actually supplied are [4,16], [19/3,76/3], and weight sum 6. The estimator therefore behaved consistently with the inputs it received. The failure is operational: those inputs did not match the declared [1,3,2,4] weight policy. The intended weighted state was [5.6,19.2], [8.84,35.36], and weight sum 10.

That is the false-positive success to design against. “partial_fit returned self” proves that the call was accepted. It does not prove that the batch carried the intended weights.

Compare the correct case, boundaries, warnings and preserved state

The canonical correct execution supplied W1=[1,3] and W2=[2,4] on both calls. Before applying the second batch, the fixture captured state rather than overwriting the only evidence of the transition:

after B1:
mean_           = [2.5, 13.0]
var_            = [0.75, 3.0]
n_samples_seen_ = 4.0

After the second correctly weighted batch, the observed state was:

mean_           = [5.6, 19.2]
var_            = [8.84, 35.36]
n_samples_seen_ = 10.0
warnings        = []

Every field matched the independent manifest at 1e-12 relative and absolute tolerance.

Capturing the pre-update state is important because incremental acceptance is a state transition test. A final state alone cannot establish which batch changed the accumulator or whether an earlier batch had already diverged. This does not imply that StandardScaler promises transactional rollback; it means the test fixture should retain enough evidence to localize a mismatch.

The partition-invariance case then redistributed exactly the same row-weight pairs across three partial_fit calls. Its observed final means were [5.6,19.2]; variance was [8.839999999999998,35.36]; and accumulated weight was [10,10]. The tiny binary floating-point representation difference in the first variance was inside the declared tolerance. No warning occurred.

The scaled-weight case multiplied every weight by ten. Because both weighted mean and weighted variance normalize by the total weight, the independent manifest expects unchanged mean and variance but a weight sum multiplied by ten. The execution produced exactly that pattern: [5.6,19.2], [8.84,35.36], and [100,100]. This is a useful identity check because a validator that looks only at mean_ and var_ would miss a tenfold change in absolute weight mass. StandardScaler’s documented n_samples_seen_ semantics expose that distinction.

The zero-weight boundary appended a synthetic outlier [1_000_000,-1_000_000] with weight 0. The observed final state remained [5.6,19.2], [8.84,35.36], and weight 10. That case is bounded to this executed fixture; it should not be generalized into a claim about every unusual weight vector or every backend.

The missing-value case inserted a NaN only in Feature B of row two. The documentation says NaNs are ignored during fitting and that counts may differ by feature. The independent Feature A contract therefore remains weight 10, mean 28/5, variance 221/25; Feature B excludes the row of weight 3, leaving weight 7, mean 150/7, and variance 1664/49.

Observed:

mean_           = [5.6, 21.428571428571427]
var_            = [8.84, 33.95918367346939]
n_samples_seen_ = [10.0, 7.0]
warnings        = []

All three fields matched.

Malformed weight cardinality was the explicit failure-mode case:

StandardScaler().partial_fit(
    B1,
    sample_weight=np.array([1.0]),
)

The executed runtime raised:

ValueError:
sample_weight.shape == (1,), expected (2,)!

The fixture treats that rejection as a passing negative-boundary result. It does not assert that the exact English exception text is a stable public contract; only the exception class is part of the manifest.

Warnings were captured for every case using warnings.catch_warnings(record=True). No warnings were observed in the seven executed cases. Absence of a warning is evidence only about this fixture and runtime. It is not the release gate. In particular, the omitted-second-batch-weights case emitted no warning even though it violated the intended weighting contract.

That distinction complements, rather than copies, the evidence technique in Refonte’s Python ZIP duplicate-member integrity article. The ZIP investigation preserves every central-directory occurrence because duplicate filenames can obscure which payload was actually read; its independent manifest verifies member identity, membership and bytes. Here the analogous identity boundary is narrower: preserve every row-weight pairing and batch boundary. There are no archive members or CRCs, and importing that workflow’s occurrence semantics would add irrelevant machinery.

Build the case-by-case evidence ledger and reject incomplete evidence

The executed evidence ledger is the release artifact, not the prose assertion that “tests passed.”

An incomplete ledger must be rejected even when the final numbers look plausible. Minimum completeness means the artifact contains the fixture revision and digest, interpreter and package versions, raw synthetic rows, every batch boundary, every batch’s weight vector or explicit unweighted status, independently expected mean, variance and weight count, observed mean_, var_ and n_samples_seen_, tolerance, warnings, exceptions and the decision.

That requirement is stricter than recording only n_samples_seen_. A count of 10 does not prove that each of the four rows carried the right individual weight; many different weight allocations can share the same sum. Conversely, checking only mean_ and var_ misses the scaled-weights case, where normalized moments remain unchanged while total weight mass changes tenfold. Release evidence needs all dimensions.

This is also where state and identity should be reconciled explicitly. For every partial-fit call, an acceptance artifact should preserve an immutable batch identifier and a digest or deterministic representation of both X_batch and sample_weight_batch. The assertion is then about a tuple:

(batch identity,
 row order,
 feature values,
 sample-weight vector,
 pre-state,
 post-state)

not about a free-floating var_ value.

The principle resembles inventory completeness in Refonte’s PostgreSQL logical replication failover runbook: that article warns that one healthy observed slot is not proof that no expected slot is missing, and it separates the expected consumer inventory from observed standby state. The present workflow applies that discipline to batch weights, not database replication: every expected weighted batch must appear in the ledger before final scaler state is admissible.

Documented behavior and observation must remain distinct. The following are documented for the tested API branch: partial_fit accepts sample_weight; StandardScaler uses the population-style variance convention; NaNs are disregarded during fitting; and weighted n_samples_seen_ represents summed weights and can become feature-specific. The following are observations from this execution: the seven cases passed their fixture assertions, omitted second-batch weights produced [4,16] and [19/3,76/3] without warning, and the source SHA-256 was df726c88….

The inference is narrower: because the incorrect call succeeded and produced a coherent estimator state for the weights actually presented, API success alone cannot establish conformance to an external intended-weight manifest. That conclusion follows from the documented optional argument plus the executed mismatch; it is not a claim that scikit-learn silently corrupts valid inputs.

Repair the workflow, revalidate the invariant, and make reruns rollback-safe

The repair is not to alter StandardScaler’s statistics after the fact. Do not overwrite mean_, var_, or n_samples_seen_ with oracle values; that would mask the defect and create an estimator state with unverifiable implementation history. Repair the input workflow that lost weight identity.

A safe wrapper can require weights explicitly for every admitted batch:

def weighted_partial_fit_required(
    scaler,
    X_batch,
    sample_weight_batch,
):
    if sample_weight_batch is None:
        raise ValueError(
            "sample_weight is mandatory for this "
            "weighted incremental workflow"
        )

    if len(sample_weight_batch) != len(X_batch):
        raise ValueError(
            "sample_weight cardinality must match X_batch"
        )

    return scaler.partial_fit(
        X_batch,
        sample_weight=sample_weight_batch,
    )

This policy is stricter than StandardScaler’s own API, intentionally. The upstream API supports both weighted and unweighted fitting. The application contract in this playbook does not: once a scaler instance is admitted to the weighted incremental workflow, an omitted weight vector is a workflow error.

The repaired invariant should be revalidated from a clean scaler:

scaler = StandardScaler()

weighted_partial_fit_required(
    scaler, B1, W1
)
weighted_partial_fit_required(
    scaler, B2, W2
)

assert np.allclose(
    scaler.mean_,
    [5.6, 19.2],
    rtol=1e-12,
    atol=1e-12,
)
assert np.allclose(
    scaler.var_,
    [8.84, 35.36],
    rtol=1e-12,
    atol=1e-12,
)
assert np.allclose(
    np.atleast_1d(scaler.n_samples_seen_),
    [10.0],
    rtol=1e-12,
    atol=1e-12,
)

A failed incremental attempt should not be “rolled back” by trying to infer and subtract its contribution from the existing estimator. The acceptance-safe recovery is to reconstruct a fresh StandardScaler from the last independently retained good input ledger and replay only approved batches in order. That is an operational recommendation derived from the need for reproducible evidence; the StandardScaler API documentation does not advertise an estimator transaction log or rollback operation.

Rerun safety therefore depends on retained inputs, not mutability tricks:

approved batch ledger
        |
        v
fresh StandardScaler()
        |
        +-- replay B1 + exact W1
        |
        +-- replay B2 + exact W2
        |
        v
compare against manifest
        |
     ACCEPT

Never continue from the defective object after discovering that a batch lacked its required weights. Even if a later call happens to move the moments closer to the expected values, the accumulator’s history remains outside the approved manifest.

The replay should also repeat the negative control. A validator can regress independently of StandardScaler: for example, a refactor might accidentally replace the weighted oracle with plain np.mean and np.var. The explicit false oracle is a compact detector for that class of test-harness failure.

Operational ownership should be assignable. The producer of each batch owns row-to-weight identity. The incremental fitting component owns rejecting missing weights under the local policy. The acceptance harness owns oracle independence and state comparison. Release engineering owns retaining the fixture, exact dependency lock and report. A failed comparison has one of three initial owners: input/weight construction, acceptance fixture, or dependency/runtime change. Do not classify it as an upstream library defect until those boundaries have been reconciled.

The rerun discipline is conceptually related to, but technically distinct from, Refonte’s PostgreSQL SKIP LOCKED queue fairness article. That queue playbook deliberately separates duplicate prevention, starvation, eventual progress and retry semantics, and its published article marks proposed experiments as unexecuted when evidence has not been gathered. This StandardScaler article has a much smaller recovery boundary: deterministic replay of synthetic weighted batches into a fresh accumulator. There are no concurrent workers, locks or fairness claims to borrow.

Scikit-learn’s broader common-pitfalls guidance reinforces the general discipline of keeping preprocessing behavior consistent and preventing inappropriate data from affecting fitted preprocessing state, including StandardScaler. That guidance is broader than this test; it should not be mistaken for a weighted-incremental acceptance specification. The specification here comes from the explicit expected-case manifest.

Apply the ACCEPT / REPAIR / RERUN / HOLD release gate

The decision is not one Boolean because different failures require different actions.

For the execution recorded in this article, the library behavior under the bounded 1.8.0 fixture is ACCEPT: all correctly weighted cases, boundaries and negative controls met their declared expectations. The deliberately omitted second-batch weights case is REPAIR as an operational workflow: partial_fit succeeded, but its state correctly represented a different weighting problem than the intended manifest. The correct remediation is to require and preserve batch weights, discard the contaminated accumulator and replay from an approved ledger.

A release using a different scikit-learn version is RERUN, not automatically ACCEPT. The current stable documentation observed during research is 1.9.1, while this evidence was executed on 1.8.0. The documented contract appears consistent for the fields used here, but an execution on 1.8.0 is not evidence of numerical output from 1.9.1.

A mismatch with incomplete input evidence is HOLD, not “probably a weight bug.” That classification matters because a wrong oracle, changed batch order, feature-specific missingness, changed runtime or genuinely changed implementation can all produce variance differences. The evidence must identify which condition exists before repair is authorized.

These related investigations address different acceptance boundaries. The OneHotEncoder article tests whether a distinct raw category collapses onto a dropped encoded baseline and preserves raw identity before encoding. The SQLite article tests whether a foreign-key setting actually took effect across transaction-state boundaries. The ZIP article tests every archive-member occurrence and an independent content manifest because duplicate names can hide damaged members. None tests weighted numeric accumulation. This article’s unique release artifact is the expected-versus-observed weighted moment ledger across StandardScaler.partial_fit batches, including an API-success case that is operationally wrong because the weights did not match.

The bounded conclusion is correspondingly narrow: on the recorded CPython 3.13.5 / scikit-learn 1.8.0 / NumPy 2.3.5 fixture, correctly preserved row-weight identity produced the independently expected weighted mean, population variance and weight counts across batch partitions. Omitting the second batch’s weights also succeeded at the API level but produced the independently expected different statistics for unit weighting of that batch. That is enough evidence to reject “the call succeeded” as a weighted release gate. It is not a guarantee about every dataset, numeric dtype, sparse path, future version or malformed weight regime.

Build relevant skills with Refonte Learning

The narrow skill demonstrated here sits at the intersection of Python data work, statistical reasoning, machine-learning preprocessing, reproducible testing and operational release discipline. Refonte Learning’s current Data Science & AI program page lists Python Data Science, Statistical Modelling, Machine Learning and Predictive Modelling, EDA and Data Visualization, model optimization, problem-solving techniques and application to industry projects among its stated competencies. The page currently states a three-month period, a dedication of 12–14 hours per week, and a requirement of being engaged in bachelor’s or postgraduate studies; it also describes a potential virtual-internship component.

The page does not establish that its curriculum specifically teaches weighted incremental mean and variance across StandardScaler.partial_fit batches, independent rational-arithmetic oracles, or this ACCEPT / REPAIR / RERUN / HOLD workflow. Readers should check the exact program page for specific curriculum coverage, mentorship, certification details, internship conditions and other enrollment claims rather than infer them from this playbook.

For engineers building toward this kind of operational weighted validation, the relevant development target is not memorizing StandardScaler syntax. It is being able to separate a documented contract from an independently calculated expectation, preserve batch-and-weight identity, design a negative control, pin the runtime that produced evidence, retain pre- and post-state, and make a release decision that does not confuse successful execution with statistical correctness. The Refonte program is relevant at that broad Data Science & AI skill-domain level; readers considering it should verify the current syllabus against those specific goals before enrolling.