A scheduled retraining job can return successfully while the candidate model still contains trees fitted to an earlier target generation. That is the misleading signal to eliminate. The release question is not “Did fit() return?” It is “Which reviewed training generation does each retained tree represent?”
That distinction matters whenever a job reuses a fitted RandomForestRegressor with warm_start=True after changing its training data. The RandomForestRegressor 1.8.0 API documents RandomForestRegressor(warm_start=True) as reusing the previous solution and adding estimators rather than automatically replacing the entire fitted forest. Its public estimators_ attribute is the collection of fitted sub-estimators.
The observed laboratory run in this playbook is intentionally narrow: CPU-only CPython 3.12.14, NumPy 2.3.5, scikit-learn 1.8.0, integer random_state, and n_jobs=1. All ten diagnostic cases passed in that environment. Two old trees trained on target zero stayed at zero after an unchanged-size call using target ten; adding two trees on the new labels produced per-tree predictions 0, 0, 10, 10 and an ensemble prediction of 5. Freshly fitted candidates produced 10.
Those results are lifecycle evidence, not an accuracy claim. The operational decision is explicit: ACCEPT reviewed same-data growth or a separately validated cold rebuild; REBUILD an unintended mixed-generation forest; RERUN evidence produced in a mismatched environment; and HOLD unknown provenance or changed training parameters.
Define whether the job is growing or replacing a forest
Start with the authorized operation, not with the scheduler configuration. “Retrain the forest” is too vague for release review because it can describe two materially different postconditions.
The first is same-data forest growth. The reviewed training snapshot and training parameters stay fixed while n_estimators increases. The intended artifact contains the original fitted trees plus additional trees built under that same reviewed training contract. The scikit-learn 1.8.0 section on fitting additional trees explicitly describes warm_start=True as a way to add trees to an already fitted random-forest model.
The second is replacement on newly reviewed data. Here the application contract says every fitted tree in the candidate must belong to the new training generation. An estimator object that retains old trees fails that contract even if the second fit() call completed normally and even if the resulting predictions look plausible.
Write one of those two intentions into the run manifest before training. For example: “Grow generation G17 from 200 to 300 trees on the unchanged reviewed snapshot D42,” or “Build generation G18 as a fresh 300-tree model on reviewed snapshot D43.” A job label such as nightly_retrain cannot substitute for this statement.
Input identity is a separate prerequisite. Refonte’s feature-identity checks address whether values reach the coordinates learned for their intended features; this playbook deliberately holds X and its ordering fixed and asks a different question about fitted-state provenance.
If nobody can state which of the two operations is authorized, the correct decision is HOLD. Do not let the presence of warm_start=True choose the model lifecycle by accident.
A useful release manifest makes the authorized population concrete. Store an operation type such as GROW_SAME_DATA or REBUILD_NEW_DATA, the parent artifact identifier when growth is intended, the reviewed dataset identifier, the target-definition revision, the requested final tree count, and the approved parameter-manifest identifier. The scheduler may still contain ordinary mechanics such as retry policy or execution time, but those fields should not be allowed to redefine the model contract. This separation gives reviewers a falsifiable postcondition: either the candidate extends a named parent under the same reviewed generation, or it begins a new lineage whose first fitted state is attributable to the new snapshot.
Pin the estimator and two training generations
The fixture keeps the feature side boring so the target-generation history is visible. X is eight ordered float64 values from zero through seven, reshaped to one column. OLD is eight zeros. NEW is eight tens. The probe contains 0.0, 3.5, and 7.0. Feature count, feature meaning, row count, row order, and feature values remain unchanged across the two target generations.
The constructor starts with two trees and uses warm_start=True, bootstrap=False, random_state=17, and n_jobs=1. Fixing an integer random state is also important for the later same-data control: the scikit-learn user guide states that, when random state is set, building a random forest iteratively to a final number of trees is equivalent to building that number in one fit under the documented scenario.
Record the actual environment rather than relying on a requirements file alone. The original preparation environment was reported as CPython 3.13.5, NumPy 2.3.5 and scikit-learn 1.8.0. The independently reproduced results used here come from CPython 3.12.14 with the same NumPy and scikit-learn versions. That run establishes behavior in its recorded environment; it does not validate a release gate pinned to CPython 3.13.5. For that gate, retain the RERUN decision until matching evidence exists. The laboratory writes sys.version, sklearn.__version__, np.__version__, platform information and a SHA-256 of the source file into results.json. A documentation version is an API baseline, not a publication date.
There is no imputer, scaler, encoder or feature selector in this fixture. That is deliberate. If a production path has preprocessing, settle imputer output-schema validation and related schema identity separately before deciding whether the forest’s tree generations are acceptable.
The data fingerprint in this lab serializes X, y, and their dtype strings into JSON with sorted keys and hashes the result. It is sufficient for this controlled numeric fixture. Production provenance usually needs substantially more: immutable snapshot identifiers, preprocessing identity, sample-weight identity, schema revision, target-definition revision, code revision, dependency lock, and the approvals that make those identifiers meaningful.
Write the prediction oracle before fitting
Declare the expected diagnostic outputs before calling the second fit. An OLD-only candidate must return 0 at every probe. A two-old-plus-two-new mixture must return 5. A NEW-only candidate must return 10.
The mixture value is independent arithmetic: (0 + 0 + 10 + 10) / 4 = 5. The RandomForestRegressor prediction documentation states that regression prediction is the mean of the individual trees’ predicted targets.
These numbers are not performance targets. Constant labels intentionally remove most modeling complexity so a reviewer can see state composition. A result of 5 is diagnostically useful because it exposes the mixed forest; it is not “halfway correct.” Likewise, 10 on three probe points does not by itself prove provenance. In this fixture, provenance comes from the controlled construction history, while the oracle makes the consequences of that history immediately inspectable.
Fit the old generation and record its trees
W1 creates the initial two-tree forest and fits it on X, OLD. The expected ensemble prediction is [0.0, 0.0, 0.0]. Immediately after that fit, the laboratory copies m.estimators_ into original, retaining Python references to each fitted tree object before any subsequent fit can change the containing forest.
That public collection is the right place to inspect this particular state boundary. The RandomForestRegressor 1.8.0 API describes estimators_ as the collection of fitted sub-estimators. It also defines warm_start=True as reusing the previous solution and adding more estimators.
Record more than the final forest size. For a production ledger, capture the candidate identifier, parent candidate if any, reviewed data identifier, complete training-parameter manifest, tree ordinal range, execution environment, source revision, and the fitted tree inventory. During a same-process diagnostic, object identity with is is useful evidence that a retained tree is literally the same Python object observed before the later call.
The input digest and object references answer different questions. The digest says which serialized numeric input the manifest represents. The references show whether later collection slots still point to the original fitted objects. Neither alone establishes a complete production lineage. The value comes from binding a known fit event to the resulting tree inventory while that history is still available.
Preserve the baseline before testing a suspect second fit. Do not immediately “fix” the object and then try to reconstruct what it used to contain. Once evidence has been overwritten, a later reviewer may be left with only a scheduler log saying the job succeeded.
Refit without increasing n_estimators
W2 takes the fitted two-tree object and calls fit(X, NEW) without increasing n_estimators. In the observed run, the candidate continues to predict zero at all three probe points, both positions retain the original tree objects, and scikit-learn emits a UserWarning. The warning captured in that recorded environment states that warm-start fitting without increasing n_estimators does not fit new trees.
That observation is consistent with the documented warm-start accounting. The scikit-learn warm-start glossary says that for ensemble estimators, the relevant count represents the total learned since initialization; when an ensemble already has N estimators and a later warm-start fit requests M, it trains M - N new estimators. When M == N, that arithmetic leaves no new estimator slots to create.
Do not elevate the exact warning sentence into a cross-version interface. The laboratory captures category and message because they are useful evidence in the pinned environment. The semantic gate should rely on explicit state and provenance checks, not on a promise that future releases will preserve that wording.
W2 is an important negative control because the test itself passes while the candidate is unsafe for a NEW-only release. The expected diagnostic result is precisely that the old trees remain and still predict zero. A green test status here means “the harness successfully reproduced the unsafe lifecycle,” not “this model is approved.”
Do not confuse a returned fit call with new training
Separate four facts in the run record: the call was invoked, input validation or other fit-time work completed, warnings were emitted, and new fitted trees were constructed. Those events are not interchangeable.
The RandomForestRegressor fit documentation describes fit(X, y, sample_weight=None) as returning the fitted estimator, but the warm-start contract modifies what repeated fitting means for retained state. A successful method return therefore does not prove the application-level postcondition “all trees were rebuilt from NEW.”
This is where scheduler-level observability becomes misleading. Exit code zero can correctly report that the process completed while the release gate correctly rejects the artifact. There is no contradiction: process completion and training-generation composition are different properties.
For the replacement contract, W2 should be classified REBUILD. Preserve the warning, old tree identities, old-only predictions, environment, and requested count as evidence. Then construct a separate replacement candidate. Suppressing the warning or changing the deployment label from OLD to NEW would alter presentation, not state.
This is also why a job-history table should distinguish requested_data_generation from resulting_model_generation. The first says what the scheduler supplied to the call; the second is a property that must be established from candidate construction evidence. In W2, NEW is the requested target generation, but the retained forest is still entirely OLD. Collapsing those two fields into a single training_dataset=NEW tag creates exactly the provenance error the ledger is supposed to prevent.
Add trees on changed labels and inspect the mixture
W3 makes the more subtle mistake: it increases n_estimators from two to four and then fits the same warm-started object on X, NEW. This time new trees are created, but the old trees are not rewritten.
The controlled experiment records the following per-tree probe outputs before considering the ensemble average:
Tree ordinal | Known training generation | Probe prediction |
0 | OLD | [0, 0, 0] |
1 | OLD | [0, 0, 0] |
2 | NEW | [10, 10, 10] |
3 | NEW | [10, 10, 10] |
It also checks that the first two positions still contain the exact original Python tree objects. With regression prediction defined as the mean across tree predictions, the forest returns [5.0, 5.0, 5.0].
This candidate violates both contracts defined earlier. It is not a NEW-only replacement because half its trees come from OLD. It is not authorized same-data growth because the additional trees were fitted on a different target generation. Calling it a “four-tree retrain on NEW” erases the history that matters.
Do not reinterpret 5 as evidence that scikit-learn somehow merged the datasets into one coherent retraining population. The experiment did not concatenate OLD and NEW. It retained two fitted OLD trees and added two fitted NEW trees. The average simply exposes both generations’ contributions on this deliberately constant-target probe.
Adding more NEW trees later would change the weighting of the mixture, but it would not satisfy an exact NEW-only provenance contract while the OLD trees remain. If the release rule is “every retained tree comes from reviewed snapshot NEW,” dilution is not repair.
The operational decision is REBUILD. Keep W3’s per-tree matrix and identity evidence because they explain why the artifact is rejected more clearly than the ensemble output alone.
Compare a cold rebuild with clone followed by fit
W4 constructs a separate four-tree estimator with warm_start=False and fits it on X, NEW. The observed run produces [10.0, 10.0, 10.0] at the probe. The important property is not merely that prediction value; it is that the candidate’s construction history starts fresh on the intended training generation.
A separate object is the safer repair boundary. It lets the mixed W3 artifact remain unchanged for investigation while the replacement candidate acquires its own data manifest, parameter manifest, evidence bundle and release identifier. No in-place surgery on estimators_ is required.
W5 takes a second route: fresh = clone(m). The laboratory checks that the cloned estimator has no estimators_ attribute before fitting, then fits it on NEW and again obtains the all-ten probe output. The scikit-learn 1.8.0 clone documentation defines clone as constructing an unfitted estimator with the same parameters; its example shows a fitted attribute present on the source and absent from the clone.
An unfitted clone is not a saved fitted checkpoint
For this playbook, clone is useful because configuration crosses the boundary while fitted trees do not. W5 explicitly verifies the absence of estimators_ before the first fit of the clone. That is very different from copying or reloading a fitted artifact whose tree collection already exists.
The clone also carries the source estimator’s parameter values, including n_estimators=4 and warm_start=True. Its first fit therefore creates its initial four-tree forest from NEW in this fixture. After that, however, it is a fitted warm-start estimator again. A later scheduler call can reintroduce the same lifecycle problem unless the provenance gate is applied.
Cloning does not approve the copied parameters. If the source object has an unreviewed max_depth or another changed training setting, the new unfitted estimator faithfully carries that configuration. Parameter review must therefore happen before the clone is accepted as a rebuild candidate.
Treat W4 and W5 as positive controls for the replacement path, not as general model-quality validation. The constant target makes their expected output exact; it does not establish that an arbitrary production forest built by either route will generalize appropriately.
Use unchanged data as the legitimate growth control
A good regression test should prove that the guard does not simply ban warm starts. W6 therefore exercises the authorized scenario: grow the forest while keeping the training data and relevant parameters unchanged.
The target is now nonconstant: [0., 1., 4., 2., 8., 3., 6., 9.]. The grown model starts with two trees, then increases to six. The comparison model fits six trees in one cold call. Both use bootstrap=True, max_depth=3, random_state=17, and the same X and y.
The random-forest user guide states that with random_state set, iterative fitting to a final tree count can produce the same estimators as fitting that final count once in the documented same-data scenario. The laboratory checks that claim in a deliberately bounded way rather than broadening it into universal bit-level equivalence.
For each corresponding pair of trees, W6 compares predictions on a finite grid from 0 through 7 in 0.25 increments, leaf assignments from apply, and the stored per-tree random_state. The observed run records the aggregate comparison as True. These checks compare the six-tree candidate grown in two stages with the candidate constructed in one fit, under the recorded CPython 3.12.14, NumPy 2.3.5 and scikit-learn 1.8.0 environment.
The limit matters. Matching predictions, leaves and seeds on one finite grid does not prove that every internal byte of every possible random forest is identical, nor does it prove equality for all possible inputs. The control supports the specific same-data construction it tests. That is enough to distinguish a legitimate growth path from the changed-target mixture without claiming more than the evidence can carry.
Put a reviewed provenance gate before warm growth
The laboratory turns the intended same-data rule into a small application guard. The manifest stores data_id(X, OLD) and a parameter dictionary from get_params(deep=False) after excluding n_estimators and warm_start. Those two excluded settings are checked separately because forest size is the permitted variable and warm start must be enabled for this operation.
guarded_growth first compares the proposed data fingerprint and current parameter set with the reviewed manifest. If either differs, it raises ValueError. It then requires model.warm_start and requires the requested tree count to be strictly greater than the number of fitted estimators. Only after those preconditions pass does it update n_estimators and call fit.
W7 demonstrates the target-change refusal. A forest fitted on OLD is paired with an OLD manifest; a request to grow it using NEW fails before the guard’s own set_params or fit call. The test checks an error type of ValueError, two retained fitted trees, param_n == 2, and identity preservation for those trees.
This is the important operational pattern: refuse the transition before the state-changing training call when the proposed operation no longer matches the reviewed lifecycle contract. Then require an explicit rebuild path rather than allowing the scheduler to improvise.
Make the guard an explicit application policy
Do not misstate that policy as a universal scikit-learn prohibition. The scikit-learn warm-start glossary says warm start generally reuses fitted state while parameters change and data stays more-or-less constant, distinguishes it from partial_fit, and also notes that different but closely related datasets can be appropriate in some use cases.
The guard is therefore narrower than the library’s conceptual allowance. It says: for this application’s full-retrain contract, changed reviewed data must not enter a retained-tree growth path. Another application could intentionally authorize a related-data strategy, but that would need its own declared semantics, evidence and review. It should not be invented after an accidental mixture appears.
The sample fingerprint is also intentionally small. A production gate should bind preprocessing, schema, weights, code, environment and immutable data identity, not just JSON-serialized arrays. It is not a lock against concurrent mutation and it cannot make an arbitrary failing fit atomic. Build a separate candidate under exclusive ownership and verify postconditions after training.
Test changed parameters and invalid count transitions
W8 attacks the parameter side of the manifest. After fitting the guarded forest, the test changes max_depth from its reviewed None value to 2 and then asks the guard to grow on the original OLD data. The data fingerprint matches, but the parameter dictionary does not, so the request is held with ValueError and the fitted tree count remains two.
Be precise about what stayed unchanged. The caller modified the estimator’s top-level max_depth before invoking the guard. The guard prevents new trees from being added under that changed configuration, but it does not magically roll the parameter value back. The script explicitly restores max_depth=None before proceeding. That distinction is exactly why release evidence should record both configuration and fitted state.
W9 is the positive counterpart. After restoring the reviewed parameter value, the guard grows the same OLD-trained forest from two to four trees using OLD again. The resulting count is four and the constant-target probe remains [0.0, 0.0, 0.0]. This proves the guard has an allowed path; it is not merely a function that rejects everything.
W10 then requests a lower target count of three from the four-tree candidate. The guard rejects the transition because the requested count is not strictly larger than len(model.estimators_), leaving four fitted trees. Its condition would also reject an equal count, though the ten-case suite directly tests the decrease rather than adding a separate equal-count guarded case.
These negative controls should stay in the regression suite. A policy is easier to trust when tests demonstrate what it refuses, what remains unchanged after refusal, and what legitimate operation still succeeds. Do not reduce those cases to “exception occurred.” Check the estimator count, parameter state that matters, identities where useful, and exact expected diagnostic output.
Finally, remember that the guard is a precondition layer. Once it executes model.set_params(n_estimators=n).fit(x, y), an arbitrary exception inside fitting is not transactionally rolled back by this helper. That is another reason to train a separate release candidate rather than mutate the model currently serving requests.
Reconcile every tree with its training generation
A training-generation ledger should make the final forest explainable without guessing from its aggregate prediction. In the controlled experiment, the reconstruction is straightforward because each construction event is known.
Candidate state | Tree-generation composition | Release interpretation |
W1 | 2 × OLD | Known old baseline |
W2 | 2 × OLD | Returned fit call, no replacement; REBUILD for NEW-only intent |
W3 | 2 × OLD + 2 × NEW | Unintended mixed generation; REBUILD |
W4 | 4 × NEW | Fresh cold candidate; eligible for separate validation |
W5 | 4 × NEW | Unfitted clone then fit; eligible for separate validation |
W9 | 4 × OLD | Authorized same-data growth control |
For production, store the ledger as construction evidence, not merely as a label attached after the fact. Each growth event should identify the parent artifact, previous and resulting tree counts, the ordinal range added, reviewed training snapshot, parameter manifest, environment manifest, code revision, and decision owner. Reconcile ranges so every retained tree is accounted for exactly once.
Training-data eligibility is adjacent but separate. Refonte’s point-in-time training-data eligibility focuses on whether information was genuinely available at the historical cutoff. Both OLD and NEW could satisfy such eligibility rules and still form an unauthorized mixed forest if your release contract required a wholly NEW candidate.
Treat unknown historical provenance as unknown
Do not infer a tree’s training dataset from a few matching predictions. Two independently trained trees can agree on a probe. A current parameter dictionary also cannot reconstruct every setting under which already-retained trees were created.
If historical manifests are missing, mark that tree range or parent artifact UNKNOWN. Then HOLD the candidate rather than filling the ledger with assumptions. A separately rebuilt candidate can establish a new known lineage while the organization investigates whether older artifacts were ever released and what downstream review, if any, is needed.
This is the difference between recorded provenance and retrospective inference. Recorded provenance binds an observed construction event to an artifact at creation time. Retrospective inference can supply hypotheses and debugging evidence, but it should not silently become release history.
The ledger should also preserve boundaries between tree generations rather than only a forest-level data label. A warm-grown forest can legitimately have multiple creation events even when all events use the same reviewed dataset. Recording ranges such as 0–199: D42/run-A and 200–299: D42/run-B lets an auditor confirm that the separate events are still part of one authorized data generation. By contrast, 0–199: D42 followed by 200–299: D43 exposes the exact transition that a NEW-only rebuild contract rejects. This range-based accounting scales better than trying to infer history from prediction behavior after the fact.
Rebuild an unintended mixed-generation candidate
Once the ledger proves an unauthorized mixture, repair by creating a new candidate, not by editing the serving or rejected artifact in place. Preserve the mixed model and its evidence first. W3’s retained tree identities, per-tree predictions, source hash, versions and warning history are valuable because they explain the defect that motivated the rebuild.
The replacement sequence is simple but strict. Freeze the rejected artifact; approve the new training snapshot and parameter manifest; construct a separate unfitted candidate; fit it on the approved data; bind the resulting tree inventory to that fit event; run lifecycle checks; and then run the independent model-quality validation required for production use. Promotion happens only after those two dimensions of review are both satisfied.
Do not “repair” the mixture by deleting the first two entries of estimators_ and keeping the later pair. That creates an artifact with an unusual mutation history and bypasses the clean evidence boundary provided by a fresh candidate. The objective is not merely to make the probe read ten; it is to produce a model whose complete retained-tree history matches the release contract.
Likewise, do not repeatedly add NEW trees until the aggregate prediction approaches the fresh model. A thresholded output comparison changes the requirement from provenance to numerical closeness. If policy says every tree belongs to NEW, the clean solution is a fresh fit whose ledger begins with NEW.
After rebuilding, preserve the relationship between rejected and replacement artifacts. The repair record should say which candidate was rejected, why, which reviewed inputs and parameters define the replacement, and which artifact was ultimately promoted. If a mixed artifact had already reached production, assign a separate historical-impact review rather than inventing consequences from the laboratory. The toy experiment demonstrates a state mechanism; it contains no evidence about business impact.
A rebuilt candidate still needs normal release evaluation. The fact that its training generation is correct does not establish predictive quality. Lifecycle correctness is necessary evidence for the intended artifact, not a substitute for testing whether that artifact is fit for its actual task.
Promotion should therefore use two independent approvals. The provenance approval answers whether the artifact was built from the intended data, parameters, code and environment with a reconciled tree history. The model-quality approval answers whether the fresh candidate meets the organization’s separate predictive and risk criteria. A candidate can pass one and fail the other. Keeping those decisions independent avoids a common review shortcut in which a good evaluation result is treated as evidence that training lineage must also have been correct. Numerical success cannot repair undocumented construction history.
Decide whether to accept, rebuild, rerun or hold
Make the final lifecycle decision explicit and candidate-specific. Do not use one green “training succeeded” flag to represent environment fidelity, provenance, diagnostic coverage and predictive validation.
Decision | Evidence condition | Action |
ACCEPT | Approved same-data growth with a reconciled ledger, or a separately validated fresh rebuild on the reviewed new data | Promote only the identified candidate after normal quality gates |
REBUILD | Known old trees remain under a NEW-only contract, including W2 or W3-style state | Preserve evidence and construct a fresh candidate |
RERUN | Interpreter, NumPy, scikit-learn, source or other required environment evidence does not match the pinned review | Produce separately labeled evidence in the required environment |
HOLD | Historical provenance is unknown, training parameters changed without review, or required ownership/evidence is unresolved | Stop promotion until the ambiguity is resolved |
The lifecycle gate is deliberately separate from broader model evaluation fundamentals. That neighboring guide discusses generalization-oriented evaluation and validation; this constant-label laboratory does not.
A RERUN should never rewrite the metadata of the mismatched run. Preserve what actually happened, then create new evidence under the intended environment. If the new run diverges, investigate the difference instead of editing expected values merely to regain a passing status.
Assign model and data ownership separately
The data owner approves the training snapshot, target definition and data eligibility. The model owner approves estimator parameters and whether the operation is same-data growth or replacement. The release owner binds evidence to the exact artifact that may be promoted. If a defective artifact may have been used historically, assign a separate owner to that impact review.
One person can hold multiple roles on a small team, but the approvals should remain distinct in the record. Execution authority is not approval authority. A scheduler that can call fit() should not be allowed to redefine the training population implicitly.
The final sign-off should let another engineer answer three questions without rerunning training: What dataset and parameter manifest were approved? Which generation does every retained tree belong to? Which artifact, exactly, was accepted?
For automation, make these decision states machine-readable rather than burying them in free-form incident notes. The candidate should not move to a promotable state unless the provenance decision is one of the explicitly acceptable outcomes. PASS_10_CASES belongs to test evidence; it is not itself an ACCEPT decision. Similarly, a successful cold rebuild should not erase the REBUILD decision attached to the earlier mixed artifact. Keeping decisions attached to artifact identifiers preserves the audit trail through retries and repairs.
Turn the ten cases into retraining regression tests
Save the following source as forest_lab.py in a fresh working directory. The observed run used CPython 3.12.14, NumPy 2.3.5 and scikit-learn 1.8.0. The script itself installs nothing and contacts no production service. For an exact source-digest comparison, save it as UTF-8 with LF line endings and one final newline. Under that representation its SHA-256 is 0f394e4909393aa343e8e54417bc7394318c40456562ba5d2fa1332500fdfd92. Different line endings change that digest even when the Python statements are otherwise equivalent.
from pathlib import Path
from datetime import datetime, timezone
import hashlib, json, platform, sys, tempfile, warnings
import numpy as np
import sklearn
from sklearn.ensemble import RandomForestRegressor
from sklearn.base import clone
X = np.arange(8, dtype=np.float64).reshape(-1, 1)
OLD = np.zeros(8)
NEW = np.full(8, 10.0)
PROBE = np.array([[0.0], [3.5], [7.0]])
def require(condition, message):
if not condition:
raise AssertionError(message)
def make(n=2, warm=True):
return RandomForestRegressor(
n_estimators=n, warm_start=warm, bootstrap=False,
random_state=17, n_jobs=1,
)
def data_id(x, y):
payload = {
'X': x.tolist(), 'y': y.tolist(),
'X_dtype': str(x.dtype), 'y_dtype': str(y.dtype),
}
return hashlib.sha256(
json.dumps(payload, sort_keys=True).encode()
).hexdigest()
def param_id(model):
return {
k: v for k, v in model.get_params(deep=False).items()
if k not in ('n_estimators', 'warm_start')
}
def guarded_growth(model, x, y, n, manifest):
if (data_id(x, y) != manifest['data']
or param_id(model) != manifest['params']):
raise ValueError('Reviewed training inputs or parameters changed')
if not model.warm_start or n <= len(model.estimators_):
raise ValueError('Expected warm growth to a larger forest')
model.set_params(n_estimators=n).fit(x, y)
return model
def main():
folder = Path(tempfile.mkdtemp(
prefix='forest-evidence-', dir=Path(__file__).parent
))
report = {
'status': 'FAIL',
'started_utc': datetime.now(timezone.utc).isoformat(),
'python': sys.version,
'sklearn': sklearn.__version__,
'numpy': np.__version__,
'platform': platform.platform(),
'source_sha256': hashlib.sha256(
Path(__file__).read_bytes()
).hexdigest(),
'old_data': data_id(X, OLD),
'new_data': data_id(X, NEW),
'cases': [],
}
def record(name, observed, expected):
row = {
'id': name, 'observed': observed,
'expected': expected, 'status': 'FAIL',
}
report['cases'].append(row)
require(observed == expected, name)
row['status'] = 'PASS'
try:
m = make().fit(X, OLD)
original = list(m.estimators_)
record('W1_old_baseline', m.predict(PROBE).tolist(), [0.0] * 3)
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter('always')
m.fit(X, NEW)
report['unchanged_count_warnings'] = [
{'category': w.category.__name__, 'message': str(w.message)}
for w in caught
]
record(
'W2_same_count_changed_labels',
{
'prediction': m.predict(PROBE).tolist(),
'same_trees': all(
a is b for a, b in zip(original, m.estimators_)
),
'warning': any(
'warm-start' in str(w.message).lower() for w in caught
),
},
{'prediction': [0.0] * 3, 'same_trees': True, 'warning': True},
)
m.set_params(n_estimators=4).fit(X, NEW)
record(
'W3_mixed_generations',
{
'prediction': m.predict(PROBE).tolist(),
'tree_predictions': [
t.predict(PROBE).tolist() for t in m.estimators_
],
'old_identity': all(
a is b for a, b in zip(original, m.estimators_[:2])
),
},
{
'prediction': [5.0] * 3,
'tree_predictions': [
[0.0] * 3, [0.0] * 3, [10.0] * 3, [10.0] * 3,
],
'old_identity': True,
},
)
cold = make(4, False).fit(X, NEW)
record('W4_cold_rebuild', cold.predict(PROBE).tolist(), [10.0] * 3)
fresh = clone(m)
unfitted = not hasattr(fresh, 'estimators_')
fresh.fit(X, NEW)
record(
'W5_clone_then_fit',
{
'unfitted_before': unfitted,
'prediction': fresh.predict(PROBE).tolist(),
},
{'unfitted_before': True, 'prediction': [10.0] * 3},
)
# Nonconstant target makes the same-data comparison informative.
y = np.array([0., 1., 4., 2., 8., 3., 6., 9.])
grid = np.arange(0, 7.25, .25).reshape(-1, 1)
grown = make().set_params(bootstrap=True, max_depth=3).fit(X, y)
grown.set_params(n_estimators=6).fit(X, y)
single = make(6, False).set_params(
bootstrap=True, max_depth=3
).fit(X, y)
same = all(
np.array_equal(a.predict(grid), b.predict(grid))
and np.array_equal(a.apply(grid), b.apply(grid))
and a.random_state == b.random_state
for a, b in zip(grown.estimators_, single.estimators_)
)
record('W6_same_data_seed_control', same, True)
guarded = make().fit(X, OLD)
manifest = {'data': data_id(X, OLD), 'params': param_id(guarded)}
before = list(guarded.estimators_)
error = None
try:
guarded_growth(guarded, X, NEW, 4, manifest)
except ValueError as e:
error = type(e).__name__
record(
'W7_changed_data_held',
{
'error': error,
'n': len(guarded.estimators_),
'param_n': guarded.n_estimators,
'same_trees': all(
a is b for a, b in zip(before, guarded.estimators_)
),
},
{'error': 'ValueError', 'n': 2, 'param_n': 2, 'same_trees': True},
)
guarded.set_params(max_depth=2)
error = None
try:
guarded_growth(guarded, X, OLD, 4, manifest)
except ValueError as e:
error = type(e).__name__
record(
'W8_parameter_change_held',
{'error': error, 'n': len(guarded.estimators_)},
{'error': 'ValueError', 'n': 2},
)
guarded.set_params(max_depth=None)
guarded_growth(guarded, X, OLD, 4, manifest)
record(
'W9_reviewed_growth',
{
'n': len(guarded.estimators_),
'prediction': guarded.predict(PROBE).tolist(),
},
{'n': 4, 'prediction': [0.0] * 3},
)
error = None
try:
guarded_growth(guarded, X, OLD, 3, manifest)
except ValueError as e:
error = type(e).__name__
record(
'W10_decrease_held',
{'error': error, 'n': len(guarded.estimators_)},
{'error': 'ValueError', 'n': 4},
)
require(len(report['cases']) == 10, 'coverage')
report['status'] = 'PASS_10_CASES'
except Exception as e:
report['error'] = {'type': type(e).__name__, 'message': str(e)}
raise
finally:
report['finished_utc'] = datetime.now(timezone.utc).isoformat()
out = folder / 'results.json'
out.write_text(json.dumps(report, indent=2) + '\n')
print(out)
if __name__ == '__main__':
main()
Run it from the directory containing the source:
python forest_lab.pyA nonzero process exit status means the run did not complete successfully. Once execution reaches its try/finally block, the script attempts to write results.json in a newly created forest-evidence-* directory even if a diagnostic assertion fails. Failures during imports, earlier initialization or report writing may leave no usable report. The presence of the file alone is not a pass condition. Inspect the process status, report-level status, all ten case rows, versions and source hash together.
The observed run returned process exit code 0 and recorded PASS_10_CASES under the environment stated above. The ten passes are diagnostic coverage, not ten approved release candidates. W2 and W3 pass because they reproduce the unsafe conditions exactly as expected.
Keep harness corrections distinct from changes to the expected model behavior. A change to warning-text matching should not weaken the tree-identity or prediction checks. Record the reason for a harness change, retain the earlier source and evidence, and rerun the corrected suite under an identified environment.
For long-term use, retain all negative controls and the legitimate W6/W9 controls. Capture warning category and message as environment evidence, but gate on state and provenance. Extend the production harness with explicit collection-length checks, complete parameter manifests, immutable dataset references and artifact identifiers. Any extension changes the source hash and should create a new labelled evidence generation rather than being presented as the original ten-case script.
Most importantly, run the suite against isolated release candidates. It is not a safe pattern to load the object currently serving requests, mutate its training state for validation, and then hope to restore it. Candidate construction, evidence collection and promotion should be separate lifecycle stages.
A practical regression suite should retain the exact unsafe expectations rather than converting them into generic exception tests. W2 should continue to assert old predictions, retained object identities and warning presence in the pinned version. W3 should continue to assert the four constituent prediction arrays and old-tree identity. W7 and W8 should prove refusal before growth, while W9 proves the authorized path still works. These checks make semantic regressions visible: a future library or harness change that alters a lifecycle assumption should force investigation, not be absorbed by weakening the oracle.
Store each rerun beside, rather than on top of, earlier evidence. The source digest, environment versions, timestamps and result rows together identify one execution. If a dependency upgrade changes behavior, a reviewer should be able to compare the old and new evidence packages directly. That practice turns a reproducibility claim into inspectable records instead of a promise that the latest run is representative of every earlier release.
Build model-lifecycle judgment with the AI Engineering Program
The engineering lesson is broader than this one estimator setting: fitted state has a history, and release review should make that history explicit enough to audit. For RandomForestRegressor, that means deciding whether an operation is authorized same-data growth or a replacement, reconciling every retained tree with that decision, and refusing to infer a clean retrain from a successful function return.
That discipline also complements model deployment discipline: the artifact validated during training review must remain identifiable through release, rather than becoming interchangeable with whatever object a scheduler most recently touched. Model, dataset and parameter versioning help preserve that connection through the broader deployment workflow.
Refonte Learning’s AI Engineering Program lists a three-month duration and 12–14 hours per week. Its published curriculum includes AI model development and optimization, data engineering for AI, scaling AI systems, AI ethics and governance, and practical project work. The programme page also lists a potential internship. For readers seeking broader training in building and deploying AI systems, review the programme details and current arrangements before enrolling.
The final decision remains technical and explicit: ACCEPT reviewed same-data growth or a separately validated cold rebuild; REBUILD an unintended mixed-generation forest; RERUN mismatched evidence; and HOLD unknown provenance or changed training parameters. That is the point at which a retraining job becomes a reviewable model lifecycle rather than merely another successful call to fit().