The Index Build Failed. What Is PostgreSQL Still Enforcing?

Probability calibration often begins with a promising metric: suddenly the Brier score is lower, or the reliability curve looks closer to the diagonal. But what if that calibrator was fit or applied to the wrong model? In this synthetic support-ticket escalation example, a sigmoid calibrator appears encouraging until its recorded base-model version is compared with the deployed estimator. The mismatch turns an apparent improvement into a release-blocking defect.

This article treats the calibrator as a distinct release component tied to a specific fitted classifier. It outlines a release-validation playbook: define the calibrated probability product, freeze the base-estimator contract, separate data roles, and fit candidate mappings without touching the base model’s training. It then evaluates candidates on protected data with probabilistic scores and reliability tables while keeping the decision policy separate. Finally, it packages acceptance, abstention, rollback, and a deliberate pairing-mismatch test. This approach builds on our machine-learning model evaluation foundations, which cover broader evaluation concepts. Here, the focus is the calibrator as a versioned release artifact. The outcome is an approve, hold, or rollback decision for one specific model-calibrator pair, not a promise that calibration always improves accuracy.

Define the Probability Product You Are Releasing

First, specify exactly what is being released. In this synthetic case, the product is the combination of an already-fitted classifier, called the base model, and a post-hoc probability calibrator. Identify each component and state the output semantics before fitting or comparing any candidate mapping.

• Base model: e.g. TicketEscalationModel v1.0 (trained on escalation tickets).

• Calibrator method: Sigmoid (Platt scaling) or Isotonic (nonparametric), each a separate candidate.

• Positive class: label 1 = escalated, 0 = not escalated. (Calibrator outputs correspond to P(escalation).)

• Features & preprocessing: e.g. “priority”, “complexity”, “channel” with one-hot on channel.

• Output claim: “Probability of ticket escalation” for the given input.

• Intended population: e.g. “customer support tickets from 2024–2026 (all products, any channel)”.

• Consumers: Tier-1 support dashboard expects P(escalation) for risk ranking.

• Owners: Data scientist (base model developer), ML engineer (calibrator integrator), and support manager (decision-policy owner).

Keep these specifications in a clear acceptance contract table. For example:

The machine-learning model evaluation foundations article covers Brier loss, log loss, discrimination, threshold choices, and slice analysis in broader detail. For this release gate, defining the product means binding the calibrator to the exact estimator, preprocessing pipeline, dataset identities, and class mapping. Any statement such as “Brier score 0.12” or “accuracy at threshold 0.5” applies only to that complete, matching pair under the recorded evaluation convention.

Freeze the Complete Base Estimator Contract

Before calibrating, freeze the model contract by capturing its exact environment. Record the input schema, feature order, fitted preprocessing, estimator identity, software versions, and class order. For example, the synthetic manifest may identify Python and scikit-learn versions, a fitted logistic regression, and the feature pipeline that produced its inputs. Record the actual environment at run time rather than assuming that a moving “stable” documentation URL matches the execution environment. Include the model ID, training-data identity, feature pipeline, class mapping, random seed, and a small set of reference inputs with expected outputs. Together, these fields form an operational fingerprint.

Wrapping an already fitted estimator in FrozenEstimator prevents subsequent calls through that wrapper from refitting the underlying model, as described in the FrozenEstimator API reference. For example:

from sklearn.frozen import FrozenEstimator

frozen_model = FrozenEstimator(trained_model)  # fit() is a no-op

The FrozenEstimator API specifies that fit is a no-op, so the fitted base estimator remains unchanged. That behavior does not prove data independence, unbiased method selection, or correct deployment pairing. The CalibratedClassifierCV API reference places responsibility on the user to keep model-fitting and calibration data disjoint when an already fitted estimator is wrapped. The manifest and release tests in this playbook are local operating controls around those library semantics.

In practice, we include a minimal API check. For example, the manifest might also require a small sample inference:

sample = np.asarray([[priority, complexity, channel_phone]], dtype=float)
ref_prob = frozen_model.predict_proba(sample)[0, positive_class_index]
# Record sample, ref_prob, classes_, environment, and tolerance in the manifest.

Record ref_prob with its reference input, feature schema, class order, numeric tolerance, and environment. Before serving, rerun the reference cases and compare the results with the approved values. A discrepancy can indicate a different preprocessing artifact, software environment, estimator, or class mapping. The library does not automatically enforce these deployment identities, so the release process must.

Use more than one reference case when practical: include ordinary inputs, boundary values, and each supported categorical path. Reference cases are not a substitute for held-out evaluation; they are deterministic deployment checks for the exact preprocessing and estimator contract.

Separate Fitting, Selection and Final Evaluation

Plan data ownership before fitting or selecting any calibrator. Use one set for fitting the base model, one for fitting calibration mappings, one for choosing among candidates, and one protected set for final assessment. Preserve the unit of independence that the data requires. Independent synthetic ticket rows can use a deterministic row split, but grouped customers, repeated incidents, or temporal production data need a split that prevents related observations or future information from crossing boundaries.

Define splits as:

• Base train (historical): data already used to produce the frozen model.

• Calibrator train: new held-out data for fitting and comparing calibrators.

• Method selection: independent rows used only to compare the fitted candidate mappings.

• Final test: a reserved set (never seen by calibrator tuning) for final acceptance testing.

For clarity, document the ID ranges or date ranges for each split (and ensure no overlap). For example:

Keep the final assessment set untouched until the selected candidate and the acceptance rule are locked. The proposed thresholds in this article are synthetic design choices, not universal guidance. If the selected calibrator fails the protected final gate, the valid outcomes are hold, retain the approved pair, gather new evidence, or redesign the evaluation. Repeatedly tuning against the same final labels would convert the protected assessment set into another selection set.

Reserve Data for Calibration and Candidate Selection

In code, we might do something like:

# Synthetic deterministic split with four distinct roles
idx = np.arange(len(tickets))
train_idx = idx[:600]
calib_idx = idx[600:750]
select_idx = idx[750:875]
final_idx = idx[875:1000]

parts = [train_idx, calib_idx, select_idx, final_idx]
for i, left in enumerate(parts):
    for right in parts[i + 1:]:
        assert set(left).isdisjoint(right)

Assert that all four identifier sets are pairwise disjoint. The pipeline does not refit the base model; it uses only calib_idx to fit each candidate, select_idx to choose the candidate under the predeclared rule, and final_idx once for acceptance. Log the dataset IDs, split logic, and random seed in the release record. CalibratedClassifierCV is not a data-lineage auditor, so these boundaries require explicit tests.

Lock the Final Evaluation Before Choosing the Winner

After reserving the splits, fit each candidate on the calibration-fit subset and compare candidates on the method-selection subset. Before examining final labels, lock the winner-selection rule and the final acceptance contract. Only the selected model-calibrator pair is evaluated on final_idx. If it fails, do not switch candidates or tune thresholds against that same protected set. Hold the release or establish a new evaluation design with genuinely new evidence.

Build a Reproducible Binary Fixture

The following synthetic fixture makes the release checks reproducible. It uses a fixed generator seed, explicit row identifiers, a binary escalation label, and four disjoint data roles. Run it only in an isolated test environment. Pin the dependency set used by the lab and record the actual versions before executing the code.

import json
import platform

import numpy as np
import pandas as pd
import sklearn
from sklearn.linear_model import LogisticRegression

SEED = 42
rng = np.random.default_rng(SEED)
n = 1000

channels = rng.choice(["email", "phone"], size=n)
priority = rng.integers(1, 5, size=n)
complexity = rng.normal(0.0, 1.0, size=n)
logit = -1.0 + 0.5 priority + 0.8 complexity + 0.2 * (channels == "phone")
true_probability = 1.0 / (1.0 + np.exp(-logit))
escalated = rng.binomial(1, true_probability)

df = pd.DataFrame({
    "row_id": np.arange(n),
    "priority": priority,
    "complexity": complexity,
    "channel_phone": (channels == "phone").astype(int),
    "escalated": escalated,
})

environment = {
    "python": platform.python_version(),
    "scikit_learn": sklearn.__version__,
    "numpy": np.__version__,
    "pandas": pd.__version__,
    "seed": SEED,
}
print(json.dumps(environment, sort_keys=True))

We then split deterministically:

idx = df["row_id"].to_numpy()
train_idx = idx[:600]
calib_idx = idx[600:750]
select_idx = idx[750:875]
final_idx = idx[875:1000]

parts = [train_idx, calib_idx, select_idx, final_idx]
for i, left in enumerate(parts):
    for right in parts[i + 1:]:
        assert set(left).isdisjoint(right)

feature_names = ["priority", "complexity", "channel_phone"]
X = df[feature_names].to_numpy(dtype=float)
y = df["escalated"].to_numpy(dtype=int)
assert set(np.unique(y)).issubset({0, 1})

base_model = LogisticRegression(random_state=0, max_iter=1000)
base_model.fit(X[train_idx], y[train_idx])
assert tuple(base_model.classes_) == (0, 1)

from sklearn.frozen import FrozenEstimator
frozen_model = FrozenEstimator(base_model)
probe = frozen_model.predict_proba(X[calib_idx[:5]])
assert probe.shape == (5, 2)
assert np.isfinite(probe).all()
assert np.allclose(probe.sum(axis=1), 1.0)

Every fixture choice above, including the data-generating process, seed, sample size, split boundaries, and classifier, is synthetic. The assertions verify binary labels, two-column probability output, class order, finite probabilities, and nonoverlapping identifiers. They establish fixture integrity, not a calibration improvement. No numerical gain is claimed without actual execution evidence.

Safe execution boundary: the code is an isolated, reproducible fixture rather than production training logic. Its explicit inputs, environment capture, assertions, and cleanup expectations follow the discipline described in our reproducible data science projects guidance. Execute it with trusted local artifacts, do not load untrusted serialized models, and remove temporary fixture outputs after validation.

Fit Candidate Mappings Without Replacing the Base Model

Fit the candidate calibrators against the exact FrozenEstimator-wrapped base model. Do not retrain or replace the base estimator inside this workflow. The current CalibratedClassifierCV API supports more than the two methods compared here; sigmoid and isotonic are the scoped candidates for this fixture, not a complete inventory. The living calibration guide still contains narrative written around two methods and an ensemble=True default, while the current API reference lists temperature as another method and reports ensemble="auto". Executable code must follow the pinned API rather than smoothing over that version or documentation-scope difference.

from hashlib import sha256
from sklearn.calibration import CalibratedClassifierCV
from sklearn.metrics import brier_score_loss

BASE_MODEL_ID = "ticket-model-v1"
PREPROCESSING_ID = "ticket-features-v1"
CALIBRATION_DATA_ID = "synthetic-ticket-fixture:rows-600-749"

sig_calibrator = CalibratedClassifierCV(
    estimator=FrozenEstimator(base_model),
    method="sigmoid",
    ensemble="auto",
)
sig_calibrator.fit(X[calib_idx], y[calib_idx])

iso_calibrator = CalibratedClassifierCV(
    estimator=FrozenEstimator(base_model),
    method="isotonic",
    ensemble="auto",
)
iso_calibrator.fit(X[calib_idx], y[calib_idx])

candidates = {"sigmoid": sig_calibrator, "isotonic": iso_calibrator}
selection_scores = {}
for name, candidate in candidates.items():
    p = candidate.predict_proba(X[select_idx])[:, 1]
    selection_scores[name] = brier_score_loss(
        y[select_idx], p, pos_label=1, scale_by_half=True
    )

selected_method = min(selection_scores, key=selection_scores.get)
selected_calibrator = candidates[selected_method]

pairing_manifest = {
    "base_model_id": BASE_MODEL_ID,
    "preprocessing_id": PREPROCESSING_ID,
    "calibration_data_id": CALIBRATION_DATA_ID,
    "method": selected_method,
    "classes": tuple(int(v) for v in selected_calibrator.classes_),
    "positive_class": 1,
}
manifest_bytes = json.dumps(pairing_manifest, sort_keys=True).encode("utf-8")
pairing_manifest["manifest_sha256"] = sha256(manifest_bytes).hexdigest()

Each candidate receives the same frozen base estimator and the same calibration-fit rows. Save the pairing information beside the candidate: base-model ID and digest, preprocessing ID, calibration-data ID, method, class order, positive class, software environment, and candidate artifact digest. The calibrator is not an anonymous probability function; its validity is conditional on that complete pair.

Fit a Sigmoid Candidate and Capture Its Pairing

The sigmoid candidate is fitted with X[calib_idx] and y[calib_idx] while the wrapped base estimator remains unchanged. Its deployment record must name the exact base model, positive class, calibration dataset, and fitted candidate artifact. The later selection and final-evaluation calls use separate rows.

Save the fitted candidate with its pairing manifest rather than exporting an unlabeled mapping. The release loader should be able to identify which model score, class order, preprocessing artifact, and data snapshot produced the mapping before it permits inference.

Treat Isotonic as a Candidate, Not an Automatic Upgrade

Fit isotonic on the same calibration-fit subset and retain it as a candidate, not an assumed upgrade. The scikit-learn probability calibration guide cautions that isotonic can overfit when calibration support is small. Treat that as a qualitative warning, not a universal sufficient-sample rule. Inspect measured validation behavior, score ties, stepwise regions, and sparse probability ranges. Isotonic ties can also alter ranking metrics, so a monotonic-calibration assumption does not justify claiming that every ranking measure must remain identical.

A stepwise isotonic fit can look convincing on the rows used to fit it while remaining unstable in sparsely supported regions. Inspect the independent selection evidence, retain tied-score behavior in the record, and reject the candidate when the evidence is inconclusive.

At this point, the base model’s prediction function is unchanged: both calibrators started from the same FrozenEstimator(base_model). We confirm this by checking one control point:

sample = X[select_idx[:3]]
probs_base = frozen_model.predict_proba(sample)
probs_sig = sig_calibrator.predict_proba(sample)
probs_iso = iso_calibrator.predict_proba(sample)

assert tuple(base_model.classes_) == (0, 1)
assert tuple(sig_calibrator.classes_) == (0, 1)
assert tuple(iso_calibrator.classes_) == (0, 1)
assert np.isfinite(probs_base).all()
assert np.isfinite(probs_sig).all()
assert np.isfinite(probs_iso).all()

The feature and base-estimator inference path is unchanged. probs_sig[:, 1] and probs_iso[:, 1] are candidate mappings derived from the same frozen estimator output and class order. Selection evidence must come from select_idx. Only the chosen candidate proceeds to final_idx, and deployment must serve that candidate with the matching base model and preprocessing manifest.

Report Probabilistic Scores Without Calling Them Pure Calibration

After selecting one candidate on the method-selection subset, evaluate that specific pair once on the protected final set. Use a declared Brier convention and a complementary proper score. The brier_score_loss API reference supports explicit positive-class identification and scale_by_half; this fixture sets pos_label=1 and scale_by_half=True so the binary score is reported on the zero-to-one convention.

from sklearn.metrics import brier_score_loss, log_loss

y_final = y[final_idx]
p_final = selected_calibrator.predict_proba(X[final_idx])[:, 1]

assert np.isfinite(p_final).all()
assert ((0.0 <= p_final) & (p_final <= 1.0)).all()

brier_final = brier_score_loss(
    y_final,
    p_final,
    pos_label=1,
    scale_by_half=True,
)
log_loss_final = log_loss(y_final, p_final, labels=(0, 1))

# Compare these results with the predeclared acceptance contract.
# Do not tune against final_idx after opening its labels.

Record pos_label, class order, sample weighting, and scale_by_half with the result. Without those fields, two reported Brier values may not be comparable. The code below evaluates only the selected candidate; the unselected candidate’s final-set score is deliberately left uncomputed to preserve the final gate.

The unscaled two-class Brier formulation and the binary half-scaled convention differ by a factor of two. A release report therefore records the API parameter, label encoding, and probability column instead of comparing scores copied from incompatible conventions.

Log loss provides a complementary proper probabilistic score. Neither aggregate score isolates calibration from all other qualities of probabilistic prediction. The scikit-learn probability calibration guide explicitly warns that a lower Brier loss alone does not prove better calibration because the score also reflects refinement and uncertainty. Report the score, reliability evidence, and sample support as separate observations. Do not turn one lower number into a claim of better ranking, better calibration, or a better action policy.

• Selection-set Brier scores for sigmoid and isotonic, using the same positive class, weights, and binary zero-to-one scaling.

• Final-set Brier loss and log loss for the selected pair only, after the method and acceptance rule are locked.

• If a ranking metric is included, report it separately and interpret ties carefully. Sigmoid calibration is order-preserving in the usual binary setting, while isotonic can create ties. A thresholded confusion matrix belongs to the decision-policy review, not the probability-calibration conclusion.

The final selection therefore combines the predeclared proper-score comparison, reliability table, uncertainty, and supported slice checks. A lower aggregate loss is evidence about that score on that population. It is not, by itself, proof of uniformly improved calibration or operational value.

Build a Reliability Table With Enough Context

A text reliability table makes support visible across probability ranges. Define the bin edges before inspecting final predictions. For each bin, retain the count, mean predicted probability, observed positive fraction, and an uncertainty interval. The table is descriptive evidence for the protected evaluation population, not a guarantee that future tickets share the same relationship.

• Bin range (e.g. 0.0–0.1)

• Number of test examples in that bin

• Mean predicted probability in the bin

• Observed fraction of positives in the bin

• 95% confidence interval for the positive fraction (e.g. Wilson score or normal approximation)

For example, the table might look like this:

The values in this table are synthetic illustrations, not executed fixture output. Sparse cells produce wide intervals and weak evidence. Keep those counts visible rather than suppressing inconvenient ranges. The same fixed bin structure must be applied to the selected pair and any approved baseline comparison.

For example, a bin containing only ten observations can show a large gap between its mean probability and observed rate while still carrying a very wide interval. That is evidence of limited support, not permission to delete the row or redraw the bin edges after inspection.

Keep Sparse Bins and Uncertainty Visible

Report empty bins as Count = 0 and mark the remaining statistics as not available. For small nonempty bins, add a declared interval method such as a Wilson interval. The statistics for data science article provides broader background on sampling uncertainty. Intervals quantify uncertainty under their stated assumptions; they do not repair dependent observations, selection bias, or an unrepresentative final set.

State whether the interval is exact, Wilson, normal-approximation, bootstrap-based, or another declared method. With very small counts, the interval may be so broad that the correct review outcome is “insufficient evidence” rather than calibrated or miscalibrated.

Fix bin edges and interval methodology before opening final results. Do not change bins because one candidate looks poor in a particular range. Keep mean probability, observed fraction, count, and interval together so reviewers can distinguish apparent agreement from weak support.

Check Operational Slices Without Inventing Certainty

Repeat the table for operationally relevant, non-sensitive slices such as ticket channel or product area. Report each slice’s support and avoid universal minimum-count claims. A sparse slice can justify an “insufficient evidence” finding, while a substantial, supported failure in a critical slice can block release even when the aggregate looks favorable. Document the population definition so a future change in slice mix is detectable rather than silently absorbed.

Illustrative slices might compare email and phone tickets or established product areas. Apply the same predeclared method to every slice, show the denominator, and distinguish a supported failure from a noisy point estimate. Aggregate agreement cannot erase a substantial failure in a critical, well-supported segment.

Keep the Decision Threshold in a Separate Contract

Changing probability estimates is conceptually separate from changing the action rule. In this synthetic case, the proposed policy escalates a ticket when P(escalation) is at least 0.5, but that threshold is a decision-policy input, not a calibration property. Preserve the current rule during calibrator validation unless a separately governed threshold change has already been approved.

The scikit-learn decision-threshold guide separates probability estimation from the rule that converts scores into actions. A changed probability mapping can move cases across an unchanged numeric threshold, and a changed threshold can alter actions without changing the underlying probabilities. Evaluate the agreed policy after documenting how it was chosen, but do not claim that calibration necessarily improves accuracy, reduces workload, or preserves every previous action.

A threshold wrapper such as FixedThresholdClassifier, or an equivalent policy layer, can make the action rule explicit. Its version and selection evidence still belong to a separate decision-policy contract, even when the threshold is packaged beside the model for deployment.

Store the policy ID and threshold beside the release manifest, while keeping ownership distinct from the calibrator. Our article on connecting data science to business decisions provides additional context for that downstream consumer contract. A future threshold change needs its own selection evidence and protected evaluation rather than being smuggled into the calibration release.

Test the Deployed Pair, Including a Deliberate Mismatch

Before approval, run a negative deployment-contract test. Deliberately present the accepted calibrator manifest with a different base-model identifier, preprocessing digest, or class order. The loader must reject the pair before serving any probability. A correctly shaped numeric output is not evidence that the mapping belongs to the estimator that produced the score.

def validate_pairing(expected_manifest, loaded_manifest):
    required = (
        "base_model_id",
        "preprocessing_id",
        "method",
        "classes",
        "positive_class",
    )
    mismatches = [
        key for key in required
        if loaded_manifest.get(key) != expected_manifest.get(key)
    ]
    if mismatches:
        raise ValueError(f"Rejected model-calibrator mismatch: {mismatches}")

approved_manifest = pairing_manifest.copy()
wrong_manifest = pairing_manifest.copy()
wrong_manifest["base_model_id"] = "ticket-model-v2"
wrong_manifest["classes"] = (1, 0)

try:
    validate_pairing(approved_manifest, wrong_manifest)
except ValueError:
    pass  # Expected negative-test observation
else:
    raise AssertionError("The deployment contract accepted a mismatched pair")

The negative fixture tests identity enforcement, not whether two independently fitted systems happen to produce different predictions. A production loader should compare immutable identifiers and class order before inference. Run the exercise only with trusted artifacts in an isolated test environment. Never load an untrusted serialized model merely to demonstrate the check.

For the approved pair, run reference cases through preprocessing, the frozen base estimator, and the calibrator. Verify finite probabilities, the zero-to-one range, two-class normalization, expected class order, and recorded tolerances. The tolerance is an engineering contract for known numerical variation, not permission to accept a different estimator. A mismatch produces a hold or rollback result.

Compare approved reference probabilities within a documented numerical tolerance and fail closed when the estimator, preprocessing, or class identity differs. A tolerance handles expected floating-point variation; it must not be used to disguise an artifact mismatch.

Version Acceptance, Abstention and Rollback Together

All evidence then goes into a release packet (like a model “release version”). This packet includes:

• Base-model identity: ticket-model-v1, its artifact digest, fitted preprocessing ID, and execution environment.

• Calibrator identity: selected method, artifact digest, and calibration-data ID synthetic-ticket-fixture:rows-600-749.

• Class interpretation: class order 0, then 1, positive class 1 = escalated, and probability-column mapping.

• Evaluation evidence: selection comparison, protected final scores, reliability table, uncertainty, and supported slice checks.

• Consumer policy: separately governed policy ID and the synthetic threshold of 0.5.

• Responsible approvals: base-model owner, calibration owner, deployment owner, decision-policy owner, and reviewer.

• Rollback target: the complete previously approved model, preprocessing, calibrator, class mapping, and policy bundle.

This release packet applies the traceability principles discussed in MLOps from notebook to production. Every change receives a new identity rather than overwriting evidence in place. A changed calibration dataset, preprocessing artifact, class mapping, or consumer policy creates a new reviewable release.

Define the rollback pair before deployment: the last approved base estimator, preprocessing artifact, calibrator, class mapping, and consumer policy. If the new pair fails service or evidence checks, restore that complete approved bundle. Replacing only the calibrator file can create another unreviewed mismatch.

Never release a calibrator alone. The approval record binds it to the frozen base-model identity, data lineage, class order, evaluation evidence, and policy version. Missing or inconsistent fields produce an abstention or hold decision, even when the candidate has a better-looking aggregate score.

Monitor Mature Outcomes and Population Changes

Even after deployment, the work isn’t done. We set up ongoing monitoring to catch calibration drift or population shifts. Key monitors include:

• Label completeness: ensure escalation outcomes continue to be logged accurately (otherwise we lose reliability feedback).

• Probability distribution: track if the distribution of P(escalation) drifts significantly from historical (which may indicate data shift).

• Reliability re-checks: at regular intervals (e.g. monthly), recalc the reliability table on newly labeled tickets to verify calibration holds.

• Service-level metrics: monitor actual decision quality (e.g. fraction of escalations caught vs total) but interpret with caution.

An unlabeled score-distribution shift is an investigation signal, not proof of miscalibration. It may reflect a population change, feature-pipeline change, or service defect. Do not automatically recalibrate on unreviewed data. Wait for sufficiently mature labels, verify the new population and independence assumptions, and then establish a new calibration and evaluation record.

Also examine whether the effective sample size has changed. Repeated tickets from one incident, customer, or automated process may be correlated, so a large row count can overstate the amount of independent evidence available for a reliability review.

A proposed shadow deployment can compare service behavior, latency, manifest identity, and probability distributions without changing production decisions. Mature labels arrive later and support a separate reliability review. Service health, score drift, label completeness, and calibration evidence are distinct signals; no single monitor automatically authorizes recalibration.

Set a label-maturity rule before calculating outcome-based reliability. Until labels are complete enough for the declared population, monitor service health and population indicators without presenting them as proof that calibration has improved or deteriorated.

Run a Thirty-Day Calibration Release Rehearsal

The following 30-day schedule is illustrative. Each stage has an owner, an artifact, and a stop criterion. The process permits rejection of both candidates when neither satisfies the predeclared contract.

1. Contract inventory (Days 1–3): Owner: ML release manager. Artifact: manifest, data-role inventory, source identities, proposed acceptance rule, and rollback target. Stop if any required identity, owner, or approval is missing.

2. Isolated fitting (Days 4–10): Owner: calibration engineer. Artifact: sigmoid and isotonic candidate bundles tied to the frozen base estimator. Stop if data overlap, invalid output, class-order ambiguity, or an unpinned environment is found.

3. Evaluation review (Days 11–20): Owner: independent reviewer. Artifact: selection evidence, protected final report, reliability tables, uncertainty, and slice analysis. Hold if no candidate satisfies the predeclared contract.

4. Shadow deployment (Days 21–28): Owner: MLOps engineer. Artifact: pairing checks, reference-case comparisons, service-health evidence, latency observations, and non-decisioning shadow logs. Recover immediately if identity or service guardrails fail.

5. Final decision (Days 29–30): Owners review the complete release packet. Activate only the accepted matching pair; otherwise retain or restore the approved rollback bundle and record the hold rationale.

At every stage, the stop rule takes precedence over calendar momentum. Neither sigmoid nor isotonic must be released. An inconclusive final review leaves the previously approved pair in place and records what additional data or design change is required before another attempt.

Turn the Pairing Test Into a Data Science Project

This fixture can become a focused data science project: preserve the frozen estimator, build disjoint datasets, compare mappings, report uncertainty, enforce the manifest, and document rollback. The verified Refonte Learning Data Science & AI Program page lists three months at 12 to 14 hours per week and includes statistical modelling, Python for data science, machine learning and predictive modelling, model optimization, exploratory data analysis, and industry-relevant projects. Review the Data Science & AI Program for those foundations; this specific FrozenEstimator and calibration-release lab is not presented here as a confirmed curriculum module.

Approve the Pair, Not a Better-Looking Score

The release decision rests on the complete evidence packet, not one better-looking metric. Approve only the exact model-calibrator pair whose identities match, whose data roles are disjoint, whose selected candidate passes the protected final contract, and whose supported slices do not reveal a release-blocking failure. Otherwise hold the release or restore the approved rollback pair.

Minimum release evidence: base-model and preprocessing identities; calibrator method and artifact digest; positive-class and class-order contract; calibration, selection, and final dataset identities; declared Brier convention; reliability and slice tables with counts and uncertainty; consumer policy ID; reference case results; deliberate mismatch rejection; sign-offs; and a tested rollback target.

A probability-shaped output is not a guarantee of correctness. Calibration evidence is population- and time-bound, sparse regions can remain uncertain, and neither candidate is entitled to release. The defensible result may be approve, hold, or recover. What matters is that the decision applies to the verified pair and remains reversible.

A candidate may legitimately remain unreleased because its final result misses the proposed tolerance, a critical slice is unsupported, label maturity is inadequate, or the pairing manifest cannot be verified. Preserving that inconclusive result is part of reproducible release governance.