Replaying a Batch Can Change Your StandardScaler

Imagine training an online StandardScaler with two approved batches of examples, A and B (the only allowed training data). After calling partial_fit(A) and partial_fit(B), the scaler holds the mean/variance of A∪B. Now suppose batch B is inadvertently replayed: another partial_fit(B). Everything runs without error, but the scaler’s statistics change. This is not a bug in the code or a new model feature; it’s exactly what incremental accumulation does. The real question for the system is: Has the learned population changed? Equally important, do we have evidence to trust it?

This article shows how simply seeing partial_fit return successfully is not enough. We use a fixed numeric fixture (one float64 feature, two immutable batches) to reveal that A+B+B yields different moments than A+B, even if some outputs look similar. Since identical outputs alone cannot prove history, we rely on three evidence layers: a published source manifest of batch IDs, a ledger of applied updates, and an independent arithmetic oracle of expected counts, means and variances. Combining these, we make a bounded recovery decision: either resume (safe to continue), hold (detect an inconsistency), or rebuild (reset and reapply unique batches). The goal is a verified preprocessing state, an approved scaler candidate, without automatically releasing it. We will distinguish partial_fit (update), fit (reset), and transform (read-only) operations, link to Scikit-learn semantics and best practices, and walk through twelve test cases. Finally, we briefly connect this lab to Refonte’s Data Science & AI program foundations (Python, NumPy, scikit-learn, statistics, etc.) as a practical skill in reliable model preparation.

1. Define which observations the scaler is allowed to learn

We assume two batches represent the complete approved population of training data. Label them A and B and list each row’s unique ID to avoid confusion with feature values. For example, imagine batch A has samples ("row-0", 0.0) and ("row-2", 2.0), and B has ("row-10", 10.0) and ("row-12", 12.0). Under the system’s data contract, each ID appears exactly once in the manifest. Repeats of an ID with new values would indicate a data correction; re-delivery of the same batch ID with identical values is a duplicate update. We distinguish those cases below.

This is purely a data-identity contract, not a model or metric check. We do not evaluate predictive accuracy or model bias here. Instead, we treat the approved batches as the only sources of truth and aim to produce a scaler that accurately reflects exactly those observations. Think of this as a preprocessing-contract check rather than an automatic production release. (For more on the broader engineering contract for learned preprocessing, see From Research to Production: Deploying ML Models at Scale, which highlights the importance of keeping training metadata like sample IDs in sync with model state.)

2. Separate an update, a reset and a transformation

Scikit-learn’s StandardScaler has distinct methods for updating versus resetting its statistics. A call to partial_fit(X) treats all of X as a single mini-batch and incrementally updates the stored mean and variance (with a “population” formula, ddof=0). The internal attribute n_samples_seen_ is a scalar count of total samples that have been processed by all partial_fit calls. Importantly, n_samples_seen_ is reset by fit(X) to the number of rows fitted in this fixture but increments across successive partial_fit calls. In our fixture, no weights or missing values are used, so the count is a simple integer number of rows (not a per-feature array).

By contrast, a call to fit(X) (or fit_transform) resets the scaler: it forgets any previous data and computes new stats only from X. Finally, transform(X) applies the learned statistics to new data but does not change the scaler state at all. In other words, transform is read-only: once the scaler is fitted (by fit or partial_fit), calling transform with any X does not modify mean_, var_, or n_samples_seen_. We will confirm this property below (case C6).

Because of this design, we cannot tell by the API alone whether a successful partial_fit call used new data or a replay. A uniform random seed or the fact that no exception was thrown does not answer that question. Nor does wrapping everything in a Pipeline magically dedupe duplicate batches. (Scikit-learn pipelines ensure that each transformation is applied in order, but they do not implement any “transaction” around partial_fit or track external IDs.) In practice, the out-of-core learning documentation separates the instance stream and feature extraction (upstream) from the incremental algorithm (downstream). We do the same: we treat the API calls as black-box statistical updates and handle deduplication outside the scaler.

Interpret n_samples_seen_ under the declared data contract

In our case, each batch has two rows, so one partial_fit adds 2 to n_samples_seen_. After partial_fit(A) then partial_fit(B), n_samples_seen_ should be 4, corresponding to 4 total samples. If B is replayed (partial_fit(B) again), it becomes 6. (Because n_samples_seen_ is a scalar count in the no-missing, no-weights scenario.) Note this is just a count of “mass” processed, not a record of which rows have been seen. Under other settings (with missing data or sample weights) n_samples_seen_ might be an array or interpreted differently, but we keep to the simplest semantics here. Importantly, the count alone is not a list of IDs; it can’t distinguish different replay scenarios.

3. Publish the batch manifest and an independent arithmetic oracle

Before running any code, we establish the oracle values for all combinations of our batches A and B. Each batch is known and immutable. For clarity, here are their contents as (ID, value) pairs (ID used only for our bookkeeping, not fed into the scaler):

• A = {("row-0", 0.0), ("row-2", 2.0)}

• B = {("row-10", 10.0), ("row-12", 12.0)}

Because these are single-feature values (float64), we can compute the population mean and variance by hand before fitting any model. The required statistics for any combination of batches follow directly from sums of values and sums of squares. For example:

• Batch A has count n=2, values {0,2}: mean = 1.0, variance = 1.0, and the standardized output for a probe value x=6 is (6−1.0)/1.0=5.0.

• Batch B has count 2, values {10,12}: mean = 11.0, variance = 1.0, and probe output (6−11.0)/1.0=−5.0.

Combining A and B (concatenating all values [0,2,10,12]) gives a population of 4 samples, sum 24 and sum of squares 248. The mean is 6.0 and the population variance is (248/4)−6.0² = 26.0 (see below). Standardizing 6.0 itself yields 0.0. If B is replayed, A+B+B has 6 samples with values [0,2,10,12,10,12], sum 46, and sum of squares 492. Its mean is 23/3 ≈ 7.6667 and variance = (492/6)−(23/3)² = 209/9 ≈ 23.222. The standardized output for 6 becomes (6−7.6667)/√(209/9) = −5/√209 ≈ −0.3459. Finally, repeating the whole population (A+B+A+B) yields count 8, mean 6.0, variance 26.0 (same as A+B) and probe 0.0 again, but notice the count is wrong for four unique samples.

We summarize these in the following table (probe = scaled 6.0):

For AB, with sum = 24 and sumsq = 248, we derive the population variance as:

248/4 − (24/4)² = 26

For ABB, with sum = 46 and sumsq = 492, we derive the population variance as:

492/6 − (46/6)² = 209/9

These values serve as our ground truth oracle for the documented population-variance convention. We will compare every fitted scaler against these expectations. Note that we computed them by hand – we will not rely on a “golden” model run to get expected numbers, because that could mask any mismatch. The table above (and the calculations behind it) stand independently of the Scikit-learn output. Ensuring reproducibility of such computations is good practice (see the source-grain and data-cleaning discipline in Data Cleaning in Python).

4. Run the complete twelve-case fixture

We implement a controlled experiment in Python 3.12.14 with scikit-learn 1.8.0, NumPy 2.3.5, and exactly our batches A and B. The file scaler_replay_fixture.py below encodes the manifest and oracle, runs all scenarios, and verifies the learned statistics against expectations. It also keeps a ledger of which batch IDs were applied. We include it here verbatim as executable reference. It requires no external data; running it in the stated environment should produce a JSON report saying PASS_12_CASES. (Warnings are caught; any mismatch raises an exception.)

"""Bounded, single-process StandardScaler replay experiment; not durable exactly-once."""

from copy import deepcopy

import hashlib

import json

import math

from pathlib import Path

import platform

import tempfile

import warnings



import numpy as np

import sklearn

from sklearn.preprocessing import StandardScaler



BATCHES = {

    "A": (("row-0", 0.0), ("row-2", 2.0)),

    "B": (("row-10", 10.0), ("row-12", 12.0)),

}

EXPECTED = {

    "A": (2, 1.0, 1.0, 5.0),

    "B": (2, 11.0, 1.0, -5.0),

    "AB": (4, 6.0, 26.0, 0.0),

    "ABB": (6, 23.0 / 3.0, 209.0 / 9.0, -5.0 / math.sqrt(209.0)),

    "ABAB": (8, 6.0, 26.0, 0.0),

}

NAMES = {"C1_full_fit", "C2_partial_AB", "C3_replay_B", "C4_replay_all",

         "C5_fit_resets", "C6_transform_readonly", "C7_restore_pair",

         "C8_stale_ledger", "C9_stale_scaler", "C10_duplicate_guard",

         "C11_changed_payload", "C12_rebuild"}



def require(condition, message):

    if not condition:

        raise RuntimeError(message)



def digest(value):

    raw = json.dumps(value, sort_keys=True, separators=(",", ":"), allow_nan=False)

    return hashlib.sha256(raw.encode()).hexdigest()



def matrix(rows):

    return np.asarray([[value] for row_id, value in rows], dtype=np.float64)



def state(scaler):

    if not hasattr(scaler, "n_samples_seen_"):

        return {"fitted": False}

    require(np.ndim(scaler.n_samples_seen_) == 0, "fixture expects scalar sample count")

    return {"fitted": True, "count": int(scaler.n_samples_seen_),

            "mean": scaler.mean_.tolist(), "variance": scaler.var_.tolist(),

            "scale": scaler.scale_.tolist(), "n_features": int(scaler.n_features_in_)}



def check(scaler, key):

    n, mean, variance, probe = EXPECTED[key]

    snapshot = state(scaler)

    require(snapshot["count"] == n and snapshot["n_features"] == 1, "count/schema mismatch")

    require(np.allclose(scaler.mean_, [mean], rtol=1e-12, atol=1e-12), "mean mismatch")

    require(np.allclose(scaler.var_, [variance], rtol=1e-12, atol=1e-12), "variance mismatch")

    require(np.allclose(scaler.scale_, [math.sqrt(variance)], rtol=1e-12, atol=1e-12),

            "scale mismatch")

    transformed = scaler.transform(np.array([[6.0]], dtype=np.float64))

    require(np.allclose(transformed, [[probe]], rtol=1e-12, atol=1e-12), "probe mismatch")

    return {"state": snapshot, "probe_6": transformed.tolist()}



def train(sequence):

    scaler = StandardScaler()

    for batch_id in sequence:

        scaler.partial_fit(matrix(BATCHES[batch_id]))

    return scaler



def validate_pair(bundle):

    ledger = bundle["ledger"]

    require(set(ledger).issubset(BATCHES), "unknown batch in ledger")

    require(all(ledger[k] == digest(BATCHES[k]) for k in ledger), "content identity mismatch")

    key = "".join(sorted(ledger))

    if key:

        check(bundle["scaler"], key)

    else:

        require(state(bundle["scaler"]) == {"fitted": False}, "state without ledger")



def apply_once(bundle, batch_id, rows):

    """Copy-on-update local demonstration; not a transaction or durable checkpoint."""

    validate_pair(bundle)

    require(batch_id in BATCHES, "unknown batch identity")

    require(digest(rows) == digest(BATCHES[batch_id]), "changed payload for known identity")

    if batch_id in bundle["ledger"]:

        return bundle, "SKIP_ALREADY_RECORDED"

    candidate = deepcopy(bundle)

    candidate["scaler"].partial_fit(matrix(rows))

    candidate["ledger"][batch_id] = digest(rows)

    validate_pair(candidate)

    return candidate, "APPLIED"



folder = Path(tempfile.mkdtemp(prefix="scaler-replay-", dir=Path(__file__).parent))

report = {"status": "FAIL", "stage": "startup", "cases": {},

         "python": platform.python_version(), "platform": platform.platform(),

         "sklearn": sklearn.__version__, "numpy": np.__version__,

         "source_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),

         "manifest": BATCHES, "expected": EXPECTED,

         "scope": "finite dense float64; one feature; no weights; in-memory single-process only"}

try:

    require(sklearn.__version__ == "1.8.0", "research fixture targets scikit-learn 1.8.0")

    def record(name, scaler, expected, verdict, sequence):

        report["stage"] = name

        entry = {"observed_before_check": state(scaler), "verdict": verdict,

                 "effective_population_batches": sequence,

                 "sample_occurrences": [rid for b in sequence for rid, value in BATCHES[b]]}

        report["cases"][name] = entry

        entry.update(check(scaler, expected))

        entry["status"] = "MATCHES_EXPECTATION"



    with warnings.catch_warnings(record=True) as caught:

        warnings.simplefilter("always")

        full = StandardScaler().fit(matrix(BATCHES["A"] + BATCHES["B"]))

        record("C1_full_fit", full, "AB", "REFERENCE", ["A", "B"])

        clean = train(["A", "B"])

        record("C2_partial_AB", clean, "AB", "ACCEPT_BOUNDED_CONTRACT", ["A", "B"])

        replay = train(["A", "B", "B"])

        record("C3_replay_B", replay, "ABB", "REJECT_REPLAY", ["A", "B", "B"])

        twice = train(["A", "B", "A", "B"])

        record("C4_replay_all", twice, "ABAB", "REJECT_DUPLICATE_HISTORY", ["A", "B", "A", "B"])

        reset = train(["A"])

        reset.fit(matrix(BATCHES["B"]))

        record("C5_fit_resets", reset, "B", "REJECT_FOR_AB_CONTRACT", ["B"])

        before = state(clean)

        clean.transform(np.array([[6.0], [100.0]], dtype=np.float64))

        require(state(clean) == before, "transform mutated learned state")

        record("C6_transform_readonly", clean, "AB", "OBSERVED_STATE_UNCHANGED", ["A", "B"])



        checkpoint = {"scaler": train(["A"]), "ledger": {"A": digest(BATCHES["A"])}}

        validate_pair(checkpoint)

        restored, action = apply_once(deepcopy(checkpoint), "B", BATCHES["B"])

        require(action == "APPLIED", "restore did not apply B")

        record("C7_restore_pair", restored["scaler"], "AB", "ACCEPT_PAIRED_RESTORE", ["A", "B"])

        require(check(checkpoint["scaler"], "A")["state"]["count"] == 2, "checkpoint mutated")



        for name, broken in (

            ("C8_stale_ledger", {"scaler": train(["A", "B"]), "ledger": {"A": digest(BATCHES["A"])}},),

            ("C9_stale_scaler", {"scaler": train(["A"]), "ledger": {k: digest(v) for k, v in BATCHES.items()}},),

        ):

            report["stage"] = name

            before = state(broken["scaler"])

            entry = {"state_before": before, "ledger": broken["ledger"], "status": "INCOMPLETE"}

            report["cases"][name] = entry

            try:

                apply_once(broken, "B", BATCHES["B"])

            except RuntimeError as exc:

                entry["error"] = str(exc)

            else:

                raise RuntimeError("inconsistent pair accepted")

            require(state(broken["scaler"]) == before, "hold mutated scaler")

            entry.update(status="EXPECTED_HOLD", verdict="HOLD_INCONSISTENT_PAIR", mutation=False)



        report["stage"] = "C10_duplicate_guard"

        before = state(restored["scaler"])

        same, action = apply_once(restored, "B", BATCHES["B"])

        require(action == "SKIP_ALREADY_RECORDED" and same is restored, "duplicate was not skipped")

        require(state(same["scaler"]) == before, "skip mutated state")

        record("C10_duplicate_guard", same["scaler"], "AB", action, ["A", "B"])



        report["stage"] = "C11_changed_payload"

        entry = {"status": "INCOMPLETE", "state_before": state(restored["scaler"])}

        report["cases"]["C11_changed_payload"] = entry

        try:

            apply_once(restored, "B", (("row-10", 10.0), ("row-12", 99.0)))

        except RuntimeError as exc:

            entry["error"] = str(exc)

        else:

            raise RuntimeError("changed payload accepted")

        require(state(restored["scaler"]) == entry["state_before"], "content hold mutated scaler")

        entry.update(status="EXPECTED_HOLD", verdict="HOLD_CHANGED_CONTENT", mutation=False)



        rebuilt = {"scaler": StandardScaler(), "ledger": {}}

        for key in ("A", "B"):

            rebuilt, action = apply_once(rebuilt, key, BATCHES[key])

            require(action == "APPLIED", "rebuild did not apply once")

        record("C12_rebuild", rebuilt["scaler"], "AB", "REBUILD_FROM_APPROVED_UNIQUE_BATCHES", ["A", "B"])

        operations = {

            "C1_full_fit": ["fit(A+B)"],

            "C2_partial_AB": ["partial_fit(A)", "partial_fit(B)"],

            "C3_replay_B": ["partial_fit(A)", "partial_fit(B)", "partial_fit(B)"],

            "C4_replay_all": ["partial_fit(A)", "partial_fit(B)", "partial_fit(A)", "partial_fit(B)"],

            "C5_fit_resets": ["partial_fit(A)", "fit(B)"],

            "C6_transform_readonly": ["start from AB", "transform([[6],[100]])"],

            "C7_restore_pair": ["deepcopy trusted A checkpoint pair", "apply_once(B)"],

            "C8_stale_ledger": ["validate AB state + A ledger", "HOLD before update"],

            "C9_stale_scaler": ["validate A state + AB ledger", "HOLD before update"],

            "C10_duplicate_guard": ["validate consistent AB", "skip repeated B"],

            "C11_changed_payload": ["validate consistent AB", "reject B changed payload"],

            "C12_rebuild": ["new unfitted pair", "apply_once(A)", "apply_once(B)"],

        }

        require(set(report["cases"]) == NAMES, "case set incomplete")

        for key in NAMES:

            report["cases"][key]["operations"] = operations[key]

        report["warnings"] = [str(w.message) for w in caught]

    report["status"] = "PASS_12_CASES"

    report["stage"] = "complete"

except Exception as exc:

    report["error"] = {"type": type(exc).__name__, "message": str(exc)}

    raise

finally:

    destination = folder / "results.json"

    destination.write_text(json.dumps(report, indent=2, allow_nan=False) + "\n")

    print(json.dumps({"status": report["status"], "cases": len(report["cases"]),

                      "report": str(destination)}, indent=2))

This script does the following 12 cases (each checking against our oracle):

• C1_full_fit: fit(A+B) as a one-shot baseline (should match “AB” stats).

• C2_partial_AB: partial_fit(A) then partial_fit(B) (incremental, also “AB” stats) – this is our reference correct incremental build (status ACCEPT_BOUNDED_CONTRACT).

• C3_replay_B: partial_fit(A), then partial_fit(B), then partial_fit(B) again. The final state is treated as “ABB” (count=6) and is flagged REJECT_REPLAY.

• C4_replay_all: partial_fit(A), partial_fit(B), partial_fit(A), partial_fit(B). This repeats the whole sequence twice (state “ABAB”, count=8) – REJECT_DUPLICATE_HISTORY.

• C5_fit_resets: partial_fit(A), then fit(B). The final fit(B) resets to B’s stats only (count=2) – not a repair of AB, so REJECT_FOR_AB_CONTRACT.

• C6_transform_readonly: After a clean AB state, call transform([...]). We check that state does not change; verdict OBSERVED_STATE_UNCHANGED.

• C7_restore_pair: Start from a trusted checkpoint pair of (scaler trained on A, ledger={"A"}). Then apply_once B. This yields AB stats and is marked ACCEPT_PAIRED_RESTORE, while the original checkpoint is unchanged.

• C8_stale_ledger: Provide an AB-trained scaler but a ledger {"A": digest(A)} missing B. The guard raises an exception (HOLD_INCONSISTENT_PAIR) and scaler is left untouched.

• C9_stale_scaler: Provide a scaler fitted only on A but a ledger claiming {"A","B"}. Again it fails the pre-check (HOLD_INCONSISTENT_PAIR).

• C10_duplicate_guard: With a consistent AB state, call apply_once with B again. The guard sees that B is already in the ledger and returns "SKIP_ALREADY_RECORDED" with state unchanged.

• C11_changed_payload: Using the AB state and ledger, try apply_once with batch id "B" but a different payload (row-12’s value changed). The digest mismatch triggers RuntimeError (HOLD_CHANGED_CONTENT).

• C12_rebuild: Start with an empty scaler and empty ledger, then apply_once(A) and apply_once(B) once each. This successfully rebuilds the correct AB state (REBUILD_FROM_APPROVED_UNIQUE_BATCHES).

The numeric cases record the observed count, mean, variance and probe output. The hold cases record the state before rejection and the caught error, then verify that the scaler remains unchanged. The script writes the detailed JSON report to results.json and prints a short summary. In the stated environment, all twelve cases passed their checks with status PASS_12_CASES and no warnings.

5. Reproduce a replay that changes the learned statistics

Now let’s focus on the cases that demonstrate the core issue. First, C1_full_fit and C2_partial_AB both produce the correct AB statistics (count=4, mean=6, variance=26) by design. fit(A+B) and partial_fit(A)+partial_fit(B) are equivalent on this data (no weights, no missing, ddof=0). They serve as our reference.

Next, C3_replay_B performs partial_fit(A), then partial_fit(B), then again partial_fit(B). This is a literal batch replay at the scaler, with no errors. The result has count=6, mean≈7.6667, variance≈23.222, and probe≈−0.3459 (see table above). All values are valid and finite. This is purely documented behavior: each call’s batch is processed, so applying B twice doubles its contribution. The code partial_fit is working as intended. However, the resulting scaler now reflects a different population (we effectively learned A+B+B). This is a mismatch to our approved data (which has only A+B).

Importantly, the scaled output shape and type are unchanged. One might naively test the scaler by sending in a sample (like x=6) and see “almost zero” difference, and conclude it’s fine. But in C3 the probe changed from 0.0 (in AB) to ≈−0.3459, so it did change (the next case shows why matching probes still cannot prove history). In general, even if the probe had remained 0, we’d still need to trust our oracle count, not just a single test value.

Repeat the entire population and expose the unchanged-moments trap

In C4_replay_all we actually double both batches ([A, B, A, B]). The final statistics are count=8, mean=6, variance=26 – numerically the same mean and variance as with count=4 (AB). The scaled output for x=6 is still 0.0. In other words, the standardization formula “collapsed” the fact of double-counting: doubling every sample made no visible change in mean or variance. This is a trap: relying on matching summary stats or probes alone can hide an extra history of updates. In C4, only by checking n_samples_seen_ or the ledger do we know the count=8 is wrong. Hence our solution keeps an external occurrence ledger and count. A matching probe (or even multiple probes) is insufficient to prove the internal state corresponds to exactly one copy of each approved sample. We must trust the ledger and the explicit count to decide.

6. Show why reset and transform are different recovery choices

Other recovery ideas might come to mind: what if we “reset” the scaler or simply apply transform again?

In C5_fit_resets, we did partial_fit(A), then fit(B). The final state is just batch B (count=2, mean=11, variance=1). That is not a valid state for AB; we have effectively thrown away A. A plain fit(B) after seeing A is not a repair, it’s a reset to a wrong population. We label it REJECT_FOR_AB_CONTRACT. In general, doing a fresh fit on a subset without replaying missing batches violates the intent that the scaler must reflect A+B.

In C6_transform_readonly, we confirmed that calling transform(X) (e.g. on [[6],[100]]) does not alter the scaler. We captured the state before and after; they are identical. (require(state(clean) == before) in the code). This matches Scikit-learn’s design: transform uses stored mean_ and var_ but does not modify them. It’s a read-only action. Thus, if we want to verify versus a “frozen-encoder” input contract, the before-and-after state check confirms that transform leaves the learned state unchanged; it cannot “fix” anything. We cannot call fit_transform at inference to auto-fix a wrong state; that would leak test data.

7. Restore learned state and source progress as a matched pair

Now we simulate an ideal recovery: suppose we have a trusted checkpoint representing exactly batch A processed and recorded. C7_restore_pair does this. We start with a dictionary {"scaler": scaler_A, "ledger": {"A": digest(A)}}, where scaler_A = train(["A"]). We verify that pair, then call our apply_once function to apply batch B. The result is a new scaler_AB and ledger {"A","B"}, both correct for the full AB population. The original scaler_A was deep-copied, so it remains unmodified. This case is marked ACCEPT_PAIRED_RESTORE. It demonstrates that if you have a proven checkpoint and you know exactly which batch to apply next, you can recover the full state.

Crucially, we require that scaler and ledger agree: they must describe the same history. If you restore only the scaler or only the ledger, you risk mismatch. (For example, “scaler has seen A and B but ledger only says A” or vice versa are inconsistent.) A genuine production system would persist both the state and a progress marker together. We do not test any on-disk serialization here; our restore uses an in-memory deepcopy. (For more on the need for consistent artifact, data and version identity, see Scikit-learn’s model persistence guide – note that joblib/pickle require the same code environment.) In practice, building a durable recovery would involve writing a checkpoint that includes both the scaler object and the source offset or batch IDs, then reloading them transactionally. Here, we treat the local deep copy as a demonstration of isolation and correctness, not a complete distributed-recovery solution.

A deepcopy demonstrates local isolation, not durable recovery

It bears repeating: our "checkpoint" in C7 was an in-memory Python deepcopy of a trusted pair. We have not tested any file-based reload or atomic commit. In a real system, a crash or concurrent write between updating data and saving the scaler could break consistency. Ensuring exactly-once behavior with brokers or databases is nontrivial. Our role here is to verify the logic that if scaler+ledger are synced, replaying the missing batch yields a correct AB state. Actual production recovery would need external evidence (e.g. Kafka offsets, persistent watermark, etc.) to prove what was truly committed.

8. Hold mismatched state and ledgers before another update

What about other bad cases? We simulate two mismatch scenarios and confirm we should not proceed:

• C8_stale_ledger: The scaler is trained on A+B (AB), but the ledger only records {"A": digest(A)}. The ledger is missing B. In apply_once, validate_pair first verifies the recorded A digest, derives the key "A", and compares A’s expected count of 2 with the scaler’s observed count of 4. It raises RuntimeError("count/schema mismatch") before incoming B is validated or applied. The fixture catches the error and verifies that the scaler’s state remains unchanged. We label this HOLD_INCONSISTENT_PAIR with no mutation.

• C9_stale_scaler: The ledger claims {"A","B"}, but the scaler object was only trained on A (count=2). Again, validate_pair detects the inconsistency: the recorded digests are valid, but the ledger’s AB population requires a count of 4 while the scaler’s observed count is 2. It raises RuntimeError("count/schema mismatch"). Again, we hold and do not change the scaler.

In both cases, we see that attempting to apply B after these mismatches is not safe. The code’s pre-check stops it. We do not automatically retry or guess what to do; we simply flag the pair as inconsistent and leave the system in a “held” state. As a policy, if neither the scaler nor the ledger can be trusted, the safe action is to HOLD and await manual inspection or rebuild from raw logs. This matches the intuition that no partial evidence (a few moments or hashes) can prove full provenance when something is already off.

9. Skip a verified duplicate without hiding changed input

What if the batch really is a duplicate of one we’ve already applied? Our system can detect that too.

In C10_duplicate_guard, after a correct AB state and ledger, we call apply_once on batch B again with the same payload. The code sees that batch "B" is already in the ledger with the matching digest, so it returns immediately with action "SKIP_ALREADY_RECORDED" and leaves the state unchanged. We consider this ACCEPTABLE as a no-op. It’s an application-layer policy: scikit-learn itself has no built-in duplicate check, so our wrapper must do it. Importantly, the guard does not pretend the batch never existed; it explicitly compares payload digests. This is not a generic streaming dedup service – it only recognizes known IDs in our manifest. In essence, it says “we’ve already incorporated this exact batch, so just move on.”

Keep a stable batch identity bound to its approved payload

Now consider C11_changed_payload: the batch ID is "B" but one of the values is changed. We attempt apply_once with (row-10=10.0, row-12=99.0) under ID "B". The digest(rows) check fails (99.0 produces a different hash than the original), so we raise an error. This case is marked HOLD_CHANGED_CONTENT. We must not skip it (because the content isn’t the one we approved) and not accept it as new (because it claims to be B). We simply hold the state as before and report the content mismatch. This enforces the idea that our identity binding “ID B → exactly this payload” must remain stable. If the source data were corrected under the same ID, the appropriate action would be an explicit revision (e.g. ingest a new batch version with a new ID). That policy is outside our experiment’s scope. For this lab, we keep the rule: match both ID and payload.

10. Rebuild a clean candidate from approved unique batches

Finally, C12_rebuild shows how to rebuild a scaler from scratch using approved batches A and B exactly once each. We start with scaler = StandardScaler() and an empty ledger {}. We apply_once A then B. Both times the action is "APPLIED". The resulting scaler has count=4, mean=6, variance=26 (the correct AB state) and the ledger {"A","B"}. We label this REBUILD_FROM_APPROVED_UNIQUE_BATCHES. This is our fallback: if we have the trusted raw batches A and B and want to be sure, we just retrain incrementally.

Notice that the code did not go back and decrement n_samples_seen_ or magically remove the effect of the extra B in case C3. We did not attempt any reverse operation. Instead, we keep the C3 scaler around (for evidence) but create a new independent scaler. StandardScaler provides no public inverse partial_fit operation. This fixture therefore restores a trusted checkpoint or rebuilds from the approved raw batches. Therefore, having the raw approved batches is essential for rebuild.

Importantly, we do not assume downstream model compatibility. We only rebuilt preprocessing. If a predictive model had been trained on the bad scaler state, the team would need to decide: do we retrain the model, or can we trust compatibility? That’s beyond this scope. Here we only say: the scaler from rebuild matches the oracle (so it’s “clean”), but we don’t automatically connect it to any stored estimator. The cleanup of any downstream artifacts remains a separate step.

11. Capture enough evidence to qualify the recovery claim

To safely claim that we have a valid scaler state, we must record a lot of metadata about the run. At a minimum, our report should include:

• Environment and versions: Python 3.12.14, scikit-learn 1.8.0, NumPy 2.3.5, OS/platform.

• Source manifest: the approved batches A and B with their row IDs and values, as recorded in the manifest field; retain their calculated payload digests alongside the ledger.

• Batch digests and ledger: retain the applied IDs and their payload hashes with the scaler checkpoint. The fixture records the inconsistent ledgers in C8 and C9.

• Operations sequence: the exact calls that were done (we log it in operations).

• Effective occurrences: the list of sample IDs actually seen in each scenario (helps count duplicates).

• Learned attributes: the scaler’s mean_, var_, n_samples_seen_, and probe values.

• Errors or warnings: any caught exceptions or warnings (our fixture collects them).

• Report status: e.g. “PASS_12_CASES” or which cases failed/held.

Collecting this evidence aligns with streaming data practices: think of the scaler as a consumer of a source. We should checkpoint both the scaler state and the source offset at once, or otherwise ensure we can recover together. See discussion in Real-Time Data Engineering: streaming integration and replay foundations.

What we have not tested: acknowledgements to the source, writing to durable storage, handling of concurrent updates or multiple sinks, or what happens if the process crashes mid-update. We assume a single-process context. A real production system would need to define a transaction boundary: e.g. “only after I have both written the scaler and updated my checkpoint do I commit that I’ve consumed batch B.” If that guarantee is missing, our logic still says: if in doubt, hold and rebuild. We do not claim an exactly-once guarantee; our skip/hold logic is a pragmatic app-layer check, not a magic protocol.

12. Turn the twelve cases into a recovery decision matrix

The table below summarizes each scenario and whether the resulting scaler satisfies the bounded AB contract. We separate the experiment outcome from the recovery decision. Approval of a scaler candidate leaves downstream model compatibility and production release as separate decisions.

“Acceptable” here means the scaler state is a valid AB fit under our contract. Negative outcomes (C3, C4, C5, C8, C9, C11) are not acceptable; they are either expected errors or inconsistencies. Cases C2, C6, C7, C10, C12 yield a valid AB scaler (though C6 and C10 are trivial no-ops). C1 is the gold standard reference. Any release of this scaler requires the exact context (same data, same config, same code version) shown here.

13. Assign ownership and keep adjacent validation problems separate

In a team setting, we divide responsibilities. The data owner ensures the batch identities and payloads are correct and declares any revisions. The ML owner (model owner) approves the preprocessing logic and decides if a recovered scaler is suitable for downstream models. The platform/job owner manages publishing of checkpoints and progress. The auditor/reviewer checks the evidence (counts, digests, errors) and chooses whether to proceed.

Note that we have not addressed upstream eligibility of the data beyond stating A and B are approved (this relates to point-in-time correctness). We assume the temporal cutoffs and labels are fine. We also do not deal with model quality metrics or complex features like encoding; those are separate problems. For example, adding weights, missing values or new features would require revisiting this whole analysis. Our contract is: given fixed features and data, only the update history may differ.

If anything beyond this exact scope changes (e.g. we switch to a weighted scaler or add features), we must deliberately revisit the validation strategy. Until then, we treat these 12 cases and their logic as domain-limited proof. Any residual uncertainty (such as how to handle truly out-of-order late data, or how to guarantee a checkpoint write) is noted but not resolved here.

14. Build the data-science foundations behind the checks

This experiment exercises practical data science skills: we used Python code and NumPy/scikit-learn to compute statistics and test a real API, all within a precise floating-point tolerance. We applied sound statistical formulas (means and variances) and understood incremental algorithms, as covered in a rigorous Data Science & AI curriculum. Refonte’s Data Science & AI Program (3 months, 12–14 hours/week) teaches exactly these skills: Python programming, numerical data handling, statistical modelling, and scikit-learn machine learning. Graduates of the program get hands-on practice building reproducible ML pipelines under mentorship and earn a certificate. (See Data Science & AI for details.)

In summary, we learned to link an approved training population to an inspectable scaler state, and to make a clear resume/hold/rebuild decision for each update scenario. These practices build confidence that a deployed model isn’t silently drifting when data delivery gets retried.