A maintainer configures CountVectorizer(lowercase=True, stop_words=['and'], ngram_range=(1, 2)), adds a callable that only performs whitespace splitting, and expects lowercase words, no and, and adjacent bigrams. The resulting matrix is valid, but its features are uppercase unigrams and still include and. That is not a hypothetical library bug or a customer incident. It is an authored demonstration of a maintenance question that deserves a literal answer: did the callable replace only token splitting, or did it replace the ordinary word feature-producing chain?
In scikit-learn 1.8.0, those are different extension points. A callable supplied as tokenizer in word mode participates inside the ordinary word-analysis path. A callable supplied as analyzer supplies the feature sequence itself; the versioned API limits several ordinary word-stage settings to other analyzer modes, and the user guide says a custom analyzer may need to reproduce stop-word filtering and n-gram generation itself. Decoding remains outside that replacement boundary. CountVectorizer 1.8.0 documentation and the 1.8 feature-extraction guide state those responsibilities explicitly.
The laboratory below therefore does not accept a configuration merely because get_params() records the intended settings or fit_transform() returns a matrix. It checks independently authored token sequences, feature names, counts, warnings, sentinel callbacks, one-setting-at-a-time controls, decoding order and a held-out transform. The supplied reference run emits warnings in the deliberately unsuitable callable cases. Its conclusion is bounded to this feature contract and this reference environment; it says nothing about universal linguistic correctness, prediction quality or compatibility with an existing downstream model.
State the feature policy before choosing the hook
The fixture has two fitting documents and one held-out document:
D1 = "CAT and DOG"
D2 = "DOG and CAT CAT"
Q1 = "DOG AND CAT"
The policy is deliberately synthetic. First lowercase. Then split on whitespace. Then remove the exact token and. Finally emit unigrams followed by adjacent bigrams over the words that remain. Choosing whether that is a good language policy is a separate problem from proving that an implementation executes it. Broader material on text preprocessing and NLP representations covers the wider conceptual context; this article stays at the narrower execution boundary.
The distinction matters because the same surface configuration can express both applicable and inapplicable options. scikit-learn documents tokenizer as replacing the string-tokenization step while preserving the surrounding word pipeline, and only when analyzer == 'word'. A callable analyzer, by contrast, is the feature extractor.
Write an independent token and count oracle
The expected policy output is authored before looking at vectorizer output:
D1 must produce ['cat', 'dog', 'cat dog'].D2 must produce ['dog', 'cat', 'cat', 'dog cat', 'cat cat'].The sorted feature coordinates must be:
['cat', 'cat cat', 'cat dog', 'dog', 'dog cat']and the training count rows must be:
D1 -> [1, 0, 1, 1, 0]
D2 -> [2, 1, 0, 1, 1].The ordering is intentional. Because and is removed before bigrams are formed, D1 contains cat dog; the bigram is adjacent in the filtered token sequence, not in the original source string.
Those literals appear in TOKENS, NAMES and COUNTS below. They are the oracle. full_features is not used to generate expected answers; it is another implementation under test. Python's str.lower() returns a lowercased copy, while str.split() with no explicit separator performs whitespace-oriented splitting rather than lowercasing anything by itself. Python 3.12 built-in types documents those operations separately.
Map the tokenizer and analyzer responsibilities
The operational difference can be summarized without treating every constructor parameter as globally enabled or disabled.
Construction | Who splits? | Lowercase / stop-word / word-ngram stages in this fixture | Feature-policy ownership |
analyzer='word', tokenizer=split_words | Custom tokenizer | Eligible ordinary word stages remain in the selected path | Shared: custom splitting, built-in surrounding stages |
analyzer=split_words | Callable analyzer | Callable supplies the feature sequence; the configured ordinary word controls tested here do not wrap it | Callable owns the emitted features |
analyzer=full_features | Complete callable analyzer | Callable explicitly lowercases, splits, removes and and builds bigrams | Callable owns the complete authored policy |
The versioned API says a custom tokenizer overrides tokenization while preserving preprocessing and n-gram generation, and applies only in word mode. It says stop_words applies to word analysis and ngram_range applies when the analyzer is not callable. The user guide is even more direct: a custom analyzer replaces preprocessor and tokenizer, while stop-word filtering and n-gram extraction occur at analyzer level, so a custom analyzer may have to reproduce them.
The tagged 1.8.0 source explains the path observed in the fixture. build_analyzer() returns a partial containing only the callable analyzer and self.decode when self.analyzer is callable. For word mode, it instead builds the preprocessor and tokenizer and passes wordngrams plus stop words into analyze. analyze invokes the decoder first; only after decoding does it branch to either the callable analyzer or the ordinary preprocessor/tokenizer/ngram sequence.
That boundary does not mean a callable disables CountVectorizer as a whole. Vocabulary learning, counting and other applicable work still occur after the feature iterator is produced. Nor does this laboratory qualify every later vectorizer option.
Separate recorded settings from executed steps
get_params() answers a configuration question: what parameter values are stored on this estimator? It does not by itself prove that every recorded option participates in the selected analyzer path.
That distinction explains the search phrase “CountVectorizer lowercase ignored.” In this article, “ignored” has a narrow meaning: with the tested callable analyzer, ordinary preprocessing lowercasing is not applied around that callable. It does not mean every parameter on CountVectorizer is ignored, nor does it identify a defect. The public documentation attaches applicability conditions to preprocessor, tokenizer, stop_words and ngram_range; the tagged implementation selects a different feature-producing branch when analyzer is callable.
The acceptance evidence must therefore come from features and counts, not constructor visibility alone.
Pin a controlled runtime and preserve the evidence
The supplied reference run used Python 3.12.14 on Linux 6.18.57 x86_64/glibc 2.39 with scikit-learn 1.8.0, NumPy 2.3.5, SciPy 1.17.0, joblib 1.5.3 and threadpoolctl 3.6.0. It began at 2026-11-03T07:49:06.573008+00:00, finished at 2026-11-03T07:49:06.580939+00:00, exited with status zero and reported PASS_10_CASES. Those timestamps establish provenance, not performance.
The supplied source was 151 lines and 681 whitespace-delimited words, with SHA-256:
e312ce499c0d19c3ac182aa57a05080d49befb721559610923b9ff384a827389A source digest identifies source bytes. It is not an execution attestation. Separately, the inspected installed scikit-learn text.py was reported to match the 1.8.0 tagged source byte-for-byte, whose supplied SHA-256 is a7a67a48b0856af50224f4548f3d2cae9f4a9d0af1e37730d4aea3da95d99d65. The public tagged file shows the analyzer branch and unused-parameter warning logic used to explain the observations.
The script enforces exact versions for scikit-learn, NumPy and SciPy. Python, platform, joblib and threadpoolctl are recorded rather than rejected by equality checks. That is an important reproducibility distinction. General background on the ecosystem belongs in Python libraries for data-science work; here, versions are evidence coordinates, not a library catalogue.
Save the source unchanged as ml_vectorizer_fixture.py. To reproduce the supplied baseline, use Python 3.12.14 rather than silently treating another 3.12 patch as identical evidence:
python3.12 -m venv .venv-text
.venv-text/bin/python -m pip install scikit-learn==1.8.0 numpy==2.3.5 scipy==1.17.0 joblib==1.5.3 threadpoolctl==3.6.0
.venv-text/bin/python ml_vectorizer_fixture.py > ml_vectorizer_results.jsonThe install command is not a complete binary or transitive-dependency lock. No installation was needed for the supplied preparation run. After execution, check the exit status, parse the JSON, require all cases V01 through V10, compare the observations with the authored expectations, and review warnings. An empty, truncated or unparsable report is incomplete evidence, not a pass.
Run the complete feature-policy comparison
The fixture constructs four vectorizers that call fit_transform: the unsuitable split callable, the word-mode custom tokenizer, the complete owned analyzer and the sentinel-hook callable. Other probes use build_analyzer() or transform(). No classifier, regressor or other predictive estimator is trained.
"""Owned callable-analyzer controls; no linguistic or model-quality approval."""
import hashlib, json, platform, sys, warnings
from datetime import datetime, timezone
from pathlib import Path
import numpy as np
import scipy, sklearn, joblib, threadpoolctl
from sklearn.feature_extraction.text import CountVectorizer
IDS = ['D1', 'D2']
DOCS = ['CAT and DOG', 'DOG and CAT CAT']
RAW = [['CAT', 'and', 'DOG'], ['DOG', 'and', 'CAT', 'CAT']]
TOKENS = [['cat', 'dog', 'cat dog'], ['dog', 'cat', 'cat', 'dog cat', 'cat cat']]
NAMES = ['cat', 'cat cat', 'cat dog', 'dog', 'dog cat']
COUNTS = [[1, 0, 1, 1, 0], [2, 1, 0, 1, 1]]
def require(ok, label):
if not ok:
raise RuntimeError(label)
def split_words(text):
return text.split()
def full_features(text):
words = [word for word in text.lower().split() if word != 'and']
return words + [a + ' ' + b for a, b in zip(words, words[1:])]
def make(analyzer='word', changes):
options = dict(lowercase=True, stop_words=['and'], ngram_range=(1, 2), token_pattern=None)
if analyzer == 'word':
options['tokenizer'] = split_words
options.update(changes)
return CountVectorizer(analyzer=analyzer, options)
def tokens(vectorizer, documents=DOCS):
analyze = vectorizer.build_analyzer()
return [list(analyze(document)) for document in documents]
def fit_record(vectorizer):
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter('always')
matrix = vectorizer.fit_transform(DOCS)
return {'features': vectorizer.get_feature_names_out().tolist(),
'counts': matrix.toarray().tolist(),
'warnings': [{'category': type(w.message).__name__, 'message': str(w.message)}
for w in caught]}
def main(report):
require(sklearn.__version__ == '1.8.0' and np.__version__ == '2.3.5'
and scipy.__version__ == '1.17.0', 'Recorded package baseline required')
original = tuple(DOCS)
def record(identity, observation, decision):
report['cases'].append({'case': identity, 'observation': observation,
'decision': decision, 'status': 'PASS'})
word, custom = make(), make(split_words)
word_tokens, custom_tokens = tokens(word), tokens(custom)
require(word_tokens == TOKENS, 'Word hook token oracle')
record('V01', {'ids': IDS, 'tokens': word_tokens}, 'MATCHES_AUTHORED_FEATURE_POLICY')
require(custom_tokens == RAW and custom_tokens != TOKENS, 'Callable hook token oracle')
params = custom.get_params()
require(params['lowercase'] is True and params['stop_words'] == ['and']
and params['ngram_range'] == (1, 2), 'Configured controls preserved')
record('V02', {'tokens': custom_tokens, 'configured_lowercase': params['lowercase'],
'configured_stop_words': params['stop_words'], 'configured_ngrams': params['ngram_range']},
'REJECT_AUTHORED_FEATURE_POLICY')
bad = fit_record(custom)
require(bad['features'] == ['CAT', 'DOG', 'and']
and bad['counts'] == [[1, 1, 1], [2, 1, 1]], 'Callable count oracle')
for parameter in ['ngram_range', 'stop_words']:
require(any(parameter in w['message'] for w in bad['warnings']), 'Missing unused warning')
record('V03', bad, 'REJECT_AUTHORED_FEATURE_POLICY')
good = fit_record(word)
require(good['features'] == NAMES and good['counts'] == COUNTS
and not good['warnings'], 'Word count oracle')
record('V04', good, 'MATCHES_AUTHORED_FEATURE_POLICY')
complete = make(full_features, lowercase=False, stop_words=None, ngram_range=(1, 1))
require(tokens(complete) == TOKENS, 'Complete analyzer token oracle')
owned = fit_record(complete)
require(owned['features'] == NAMES and owned['counts'] == COUNTS
and not owned['warnings'], 'Complete analyzer count oracle')
record('V05', owned, 'MATCHES_AUTHORED_FEATURE_POLICY')
calls = []
def forbidden(text):
calls.append(text)
raise RuntimeError('Separate hook unexpectedly executed')
bypass = fit_record(make(split_words, preprocessor=forbidden, tokenizer=forbidden))
require(calls == [] and bypass['features'] == bad['features']
and bypass['counts'] == bad['counts'], 'Separate hooks bypassed')
record('V06', {'separate_hook_calls': len(calls), bypass}, 'EXPECTED_UNUSED_HOOKS')
variations = []
for name, change, expected in [
('lowercase', {'lowercase': False}, ['CAT', 'DOG', 'CAT DOG']),
('stop_words', {'stop_words': None}, ['cat', 'and', 'dog', 'cat and', 'and dog']),
('ngram_range', {'ngram_range': (1, 1)}, ['cat', 'dog']),
]:
first = tokens(make(change), DOCS[:1])[0]
second = tokens(make(split_words, **change), DOCS[:1])[0]
require(first == expected and second == RAW[0], 'Eligibility positive control')
variations.append({'parameter': name, 'word_features': first, 'callable_features': second})
record('V07', variations, 'DOCUMENTED_HOOK_BOUNDARIES_CONFIRMED')
frozen = [dict(v.vocabulary_) for v in [word, complete]]
held = ['DOG AND CAT']
matrices = [v.transform(held).toarray().tolist() for v in [word, complete]]
require(matrices == [[[1, 0, 0, 1, 1]], [[1, 0, 0, 1, 1]]]
and [v.vocabulary_ for v in [word, complete]] == frozen, 'Held document oracle')
record('V08', {'id': 'Q1', 'document': held[0], 'counts_word_and_complete': matrices,
'vocabularies_unchanged': True}, 'MATCHES_AUTHORED_FEATURE_POLICY')
received = []
def inspect_decoded(text):
received.append({'type': type(text).__name__, 'value': text})
return text.split()
analyze_bytes = CountVectorizer(analyzer=inspect_decoded, token_pattern=None,
encoding='utf-8', decode_error='strict').build_analyzer()
decoded = analyze_bytes(b'CAT and DOG')
require(decoded == RAW[0] and received == [{'type': 'str', 'value': DOCS[0]}],
'Decoding remains before callable')
record('V09', {'received': list(received), 'features': decoded}, 'DECODED_BEFORE_ANALYZER')
before = len(received)
try:
analyze_bytes(b'\xff')
except UnicodeDecodeError as error:
error_type = type(error).__name__
else:
raise RuntimeError('Invalid UTF-8 was accepted')
require(len(received) == before, 'Invalid bytes reached analyzer')
record('V10', {'error': error_type, 'analyzer_call_delta': len(received)-before},
'EXPECTED_DECODE_REJECTION')
require(tuple(DOCS) == original and len(set(IDS)) == len(DOCS), 'Source corpus changed')
require([r['case'] for r in report['cases']] == ['V%02d' % i for i in range(1, 11)],
'Case inventory')
report.update(status='PASS_10_CASES', corpus_preserved=True)
if name == '__main__':
report = {'status': 'HOLD', 'cases': [], 'started_utc': datetime.now(timezone.utc).isoformat(),
'source_sha256': hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),
'environment': {'python': sys.version, 'platform': platform.platform(),
'sklearn': sklearn.__version__, 'numpy': np.__version__, 'scipy': scipy.__version__,
'joblib': joblib.__version__, 'threadpoolctl': threadpoolctl.__version__}}
try:
main(report)
except Exception as error:
report['error'] = {'type': type(error).__name__, 'message': str(error)}
report['finished_utc'] = datetime.now(timezone.utc).isoformat()
print(json.dumps(report, indent=2, allow_nan=False))
sys.exit(0 if report['status'] == 'PASS_10_CASES' else 1)Two implementation details in that source are especially important. First, split_words() does nothing except text.split(). Python documents splitting and lowercasing as distinct string operations, so the function contains no hidden case transformation. Second, the complete analyzer explicitly writes the policy into full_features: lower(), whitespace splitting, exact-token filtering, then bigram construction. That makes ownership visible and testable rather than relying on ordinary word-mode options to wrap a callable analyzer.
The supplied reference result PASS_10_CASES means all ten assertions behaved as authored. It does not mean all tested vectorizers satisfy the desired policy. V02 and V03 are successful tests precisely because they demonstrate and reject the unsuitable split-only callable analyzer.
Read the ten cases without confusing rejection with failure
The supplied reference ledger is an acceptance record for the tests, not a blanket approval for every implementation.
Case | Reference observation | Meaning |
V01 | Word tokenizer produces the literal TOKENS sequences | Declared feature policy matches |
V02 | Callable split returns RAW despite the retained three settings | Declared feature policy rejected |
V03 | Callable split fits 3 raw features; 2 warnings observed | Count matrix exists, policy still rejected |
V04 | Word tokenizer fits NAMES and COUNTS; no warnings observed | Count oracle matches |
V05 | Owned complete analyzer fits the same names/counts; no warnings | Separate implementation matches this corpus |
V06 | Sentinel hook calls are 0; raw matrix remains; 4 warnings observed | Separate hooks were not invoked |
V07 | Three individual settings change word output, not split-callable output | Eligible-stage positive controls |
V08 | Both qualified alternatives transform Q1 to [1, 0, 0, 1, 1] | Two vocabularies unchanged across this transform |
V09 | Valid bytes arrive at the callback as one str | Decoding precedes the callable |
V10 | Invalid UTF-8 raises UnicodeDecodeError; callback delta 0 | Expected decoder rejection |
The observed behavior aligns with the documented customization boundary and the tagged source path: callable analyzers bypass the ordinary preprocessor/tokenizer/ngram branch, while decoding is still passed into the callable-analyzer wrapper.
Retain the warning evidence
V03 records two warnings naming ngram_range and stop_words. The tagged source contains explicit unused-parameter warnings for non-default ngram_range with a callable analyzer and for stop_words when the analyzer is not word mode.
V06 additionally records warnings for preprocessor and tokenizer, producing four observed warnings in that supplied run. The source also contains those warning branches. But four is an observation, not a universal warning-count contract: V06 records its warning list without asserting an exact length. V03 is stricter only about the presence of the named ngram_range and stop_words messages.
Warnings therefore support the feature evidence; they do not replace it. build_analyzer() probes also are not a complete warning audit because warning behavior may be associated with fitting or parameter validation rather than every direct analyzer call.
Reconcile feature names with the actual counts
A document-term matrix is interpretable only together with its feature coordinates. CountVectorizer exposes get_feature_names_out() for that mapping, while fit_transform() learns a vocabulary and returns the count matrix.
For the rejected callable analyzer, coordinates are:
["CAT", "DOG", "and"]and the rows are:
D1 -> [1, 1, 1]
D2 -> [2, 1, 1]Those counts are perfectly self-consistent. D1, "CAT and DOG", emits one CAT, one DOG and one and. D2, "DOG and CAT CAT", emits two CAT, one DOG and one and. The failure is not arithmetic. It is that those feature names violate the authored lowercase/remove-and/bigram policy.
For both matching alternatives, coordinates are:
["cat", "cat cat", "cat dog", "dog", "dog cat"]with:
D1 -> [1, 0, 1, 1, 0]
D2 -> [2, 1, 0, 1, 1]D2 contains cat twice, which explains coordinate zero having count 2. After filtering and, its remaining word sequence is dog, cat, cat, so the adjacent filtered-word bigrams are dog cat and cat cat, one each.
This is why “the matrix came back” is weak evidence. Both the rejected and matching implementations produce valid count matrices. Feature names plus literal expected rows answer the actual maintenance question.
Choose the smaller hook or own the complete policy
There are two qualified implementations in this fixture.
When the customization need is only splitting, the smaller hook is tokenizer=split_words while retaining analyzer='word' and setting token_pattern=None. The documented contract for a custom tokenizer is specifically to replace tokenization while preserving the surrounding preprocessing and n-gram machinery in word mode. For this corpus, V01 and V04 show that this path produces exactly TOKENS, NAMES and COUNTS.
When the application intends to own the whole feature policy, the alternative is a complete callable analyzer. full_features explicitly lowercases, splits, filters and constructs bigrams. Its vectorizer neutralizes the ordinary settings with lowercase=False, stop_words=None and ngram_range=(1, 1) so the configuration communicates where responsibility lives. V05 independently verifies its outputs against the same literal oracle.
The scikit-learn user guide supports that division of labor: a tokenizer is one stage; a custom analyzer can replace preprocessing and tokenization and may need to reproduce stop-word filtering and n-gram creation.
Removing warnings is not an acceptance criterion by itself. A warning-free implementation could still encode the wrong policy. Conversely, the deciding evidence here is that two separately constructed and separately fitted vectorizers reproduce the authored tokens, feature names and counts.
Specify adjacency after stop-word removal
D1 begins as CAT and DOG. After lowercasing and exact removal of and, the remaining sequence is cat, dog. The policy then forms the bigram cat dog.
That means “adjacent bigram” in this fixture means adjacency after filtering. It does not mean adjacency in the original character stream. The ordinary word-ngram path accepts a stop-word list and produces word n-grams from the remaining tokens; the source passes stop words into wordngrams in word mode.
If a task requires original adjacency (so that cat dog must not be formed when and appeared between the words), the contract must say so and the implementation must change. That is a language-policy decision, not a hook-selection trick. Refonte Learning's discussion of preserving text meaning during data cleaning is relevant to the review principle: transformations should be deliberate rather than assumed harmless. Neither implementation here is presented as universally preferable for NLP.
Prove that the separate hooks were not invoked
V06 turns a subtle path-selection claim into an executable observation. It supplies the callable analyzer split_words, then sets both preprocessor and tokenizer to a sentinel function named forbidden.
The sentinel has two effects: append the received value to calls, then raise RuntimeError. If either separately configured hook executes, the fit cannot quietly continue. In the supplied reference run, calls remains empty, the feature names and counts remain equal to the rejected raw callable result, and the case records separate_hook_calls: 0.
That observation matches the 1.8.0 implementation. When self.analyzer is callable, build_analyzer() returns analyze parameterized with the analyzer and decoder. It does not attach the separately configured preprocessor or tokenizer. analyze itself decodes first and then, when an analyzer is present, invokes that analyzer rather than entering the preprocessor/tokenizer/ngram branch.
The unused-parameter warnings point in the same direction. The tagged source warns that preprocessor will not be used when analyzer is callable and that tokenizer will not be used when the analyzer is not word mode.
The limit is important: V06 counts only these two instrumented sentinel hooks. It does not prove that no callbacks anywhere in the Python process execute, that preprocessing is absent from an upstream application, or that every CountVectorizer option is bypassed. It demonstrates that these separately configured preprocessor and tokenizer callables were not invoked in this fit.
Change one setting at a time as a positive control
A single bad output can be explained in many ways. V07 makes the hook boundary more informative by changing one ordinary word-stage setting at a time and comparing two analyzers on D1.
With word mode and lowercase=False, the feature sequence becomes:
["CAT", "DOG", "CAT DOG"]That demonstrates that the lowercasing setting can affect the ordinary word path. With the callable split analyzer, the corresponding result remains:
["CAT", "and", "DOG"]With word mode and stop_words=None, the features become:
["cat", "and", "dog", "cat and", "and dog"]The retained and now participates in bigrams. The corresponding callable split still emits:
["CAT", "and", "DOG"]Finally, setting ngram_range=(1, 1) in word mode yields:
["cat", "dog"]while the callable split again remains:
["CAT", "and", "DOG"].These observations track the documented applicability rules: stop_words is a word-analyzer option, ngram_range applies when the analyzer is not callable, and a custom tokenizer remains embedded in word mode.
Positive controls answer a stronger question than “did the unsuitable implementation look wrong?” They show that each setting is capable of changing output on the eligible word path while the same change does not alter the tested split-callable output. V07 uses build_analyzer() rather than fitting three more models, so it does not establish warning counts or fitted-vocabulary behavior for those probes.
Keep decoding inside the execution boundary
“Raw input” can be misleading if it is read as “bytes are handed untouched to the callable.” The CountVectorizer API exposes decode(doc) as the operation that decodes bytes or strings according to configured decoding parameters, and the tagged source calls the decoder before the callable analyzer.
V09 makes that ordering visible. It builds:
CountVectorizer(analyzer=inspect_decoded, token_pattern=None, encoding='utf-8', decode_error='strict')and then analyzes:
b'CAT and DOG'The callback records what it receives. In the supplied reference run, the record is:
{'type': 'str', 'value': 'CAT and DOG'}and the callback emits:
['CAT', 'and', 'DOG'].So the callable receives a Python str, not the original bytes. This is consistent with the API's documented decoding step and the _analyze implementation, where decoder(doc) precedes the analyzer(doc) call.
V10 tests the rejection side with b'\xff' under UTF-8 and decode_error='strict'. The supplied reference observation is UnicodeDecodeError with analyzer_call_delta equal to zero. In other words, malformed input under that configured decoder fails before one more analyzer callback is entered.
Separate decoder evidence from language validation
These two byte controls establish a very narrow ordering property for valid ASCII bytes and one invalid byte under UTF-8 strict decoding. They do not establish Unicode normalization behavior, multilingual segmentation, equivalence between lowercasing and casefolding, filename/file-object behavior, malformed-input policy beyond this case, or security properties.
That distinction also avoids an opposite overclaim. A callable analyzer replaces the ordinary feature-producing stages tested here, but it does not bypass decoding. The source explicitly describes _analyze such that, when an analyzer is supplied, only the decoder from the surrounding chain remains in use before the analyzer call.
Verify a held-out document and identify the consumer boundary
V08 uses the already fitted word-tokenizer and complete-analyzer vectorizers on the held-out authored document:
Q1 = "DOG AND CAT"Both produce:
[1, 0, 0, 1, 1]under feature coordinates:
["cat", "cat cat", "cat dog", "dog", "dog cat"].The row is easy to inspect manually. Lowercasing produces dog and cat; removing and leaves dog, cat; unigrams contribute one dog and one cat; the filtered-word bigram contributes one dog cat. Hence the five coordinates are cat=1, cat cat=0, cat dog=0, dog=1, dog cat=1.
The test also copies both vocabulary_ dictionaries before transformation and requires the two dictionaries to compare equal afterward. That establishes dictionary equality across this one transform. It does not establish whole-object immutability, serialization stability or invariance of every internal attribute.
Nor is Q1 evidence of predictive generalization: there is no target, predictive estimator or quality metric in this laboratory. It is a feature-construction acceptance probe.
That boundary matters for downstream consumers. A model trained against one feature vocabulary consumes coordinates with learned identities. Refonte Learning's separate treatment of feature identity at the model boundary demonstrates why matching width alone is not enough when coordinates change. If a real system changes analyzer policy or vocabulary, downstream compatibility requires its own review. This fixture does not demonstrate migration, retraining, rollback or compatibility with a previously fitted model.
Make a bounded accept, reject or hold decision
The result can be operationalized as a three-way decision rather than a vague “looks good.”
Decision | Condition | Evidence required here | Next action |
Match | Tested implementation reproduces the authored policy | Literal token oracle, named features, literal counts, expected cases and warnings reviewed in the qualified baseline | Accept only the fixture-level feature-policy conclusion |
Reject | Implementation produces features inconsistent with the authored policy | Example: split callable returns RAW, features CAT/DOG/and, and raw counts despite stored settings | Select a hook that actually owns the intended stages and fit it separately |
Hold | Evidence is incomplete or comparability is unresolved | Unknown policy, changed runtime without qualification, missing case, malformed report, unexpected observation or unreviewed consumer compatibility | Resolve the missing evidence before recording a match |
On the supplied reference evidence, tokenizer=split_words in word mode is a match for the authored lower/remove-and/unigram-plus-bigram policy. The separately fitted full_features callable analyzer is also a match on this corpus. The callable analyzer=split_words is a reject for that policy, even though it returns a usable matrix.
The decision remains bounded. Matching these fixtures does not establish linguistic suitability, accuracy, model compatibility or production readiness.
Classify incomplete evidence explicitly
A review should require a complete V01–V10 case inventory, expected case outcomes and warning evidence before marking the run as matching the reference acceptance record. PASS_10_CASES means the tests behaved as authored; it does not convert V02 or V03 from policy rejection into policy approval.
Likewise, no report is not a passing report. Missing, partial or unparsable JSON is hold because the evidence cannot be evaluated. The script can also fail before producing useful JSON (for example, imports or reading its own source bytes can fail), so the absence of an error object cannot be inferred from an absent file.
Do not edit expected literals until a failing implementation passes. That destroys the independence of the oracle. Do not globally suppress warnings merely to obtain cleaner output. And do not treat the script's aggregate pass as deployment authorization. Its scope is feature construction under the authored fixture.
Assign ownership for policy and future changes
A maintainable handoff should make the contract inspectable by someone who did not write the callback. At minimum, preserve one review record containing:
the named owner of the text-feature policy;
document IDs D1, D2 and Q1, together with the exact lower/split/remove-and/post-filter-bigram rule;
the independently authored TOKENS, NAMES, COUNTS and held-out expected row;
the chosen hook (word-mode tokenizer or complete callable analyzer) and the visible versioned configuration;
the source checksum and filename used for the qualification script;
the recorded Python, platform and package environment plus execution timestamps and exit status;
the complete V01–V10 inventory and each expected decision, including deliberate rejections;
captured warning evidence rather than a statement that the run was “clean”;
known downstream consumers whose feature contract may depend on vocabulary identity; and
explicit limits: what was not tested and what remains unresolved.
This is the same broader engineering principle behind preserving engineering evidence in data-science projects: an output is more reviewable when the decisions and reproducibility evidence travel with it. Here, however, the evidence concerns only CountVectorizer feature construction.
Policy ownership and implementation ownership should not be conflated. A language-policy change (such as deciding that and must remain, or that bigrams must preserve original adjacency) needs approval from whoever owns the intended feature semantics. An implementation change (such as replacing the custom tokenizer with a callable analyzer that is supposed to reproduce the same policy) needs engineering qualification against the existing oracle. Often both reviews are necessary because a code change can accidentally alter semantics, while a deliberate semantic change necessarily invalidates some old expectations.
Recovery from the rejected split-callable case is therefore not “toggle a parameter until the warning disappears.” Select one of the qualified ownership patterns, construct and fit it separately, verify its literal outputs, and then review its consumers. Do not mutate the hook of a fitted object and assume an existing model remains compatible.
Requalification should be triggered by a changed intended policy, a changed analyzer or tokenizer implementation, a relevant library baseline change, or a changed downstream consumer contract. Old reports remain useful as historical comparison points, but they are not evidence that the new configuration behaves identically.
Several CountVectorizer controls remain explicitly outside this qualification: nondefault min_df, max_df and max_features; binary counting; fixed vocabularies; persistence and reload behavior; and multilingual corpora. The API documents these as separate vocabulary or counting controls, but this fixture does not exercise them. It also does not qualify punctuation behavior beyond the supplied whitespace-only strings, general Unicode handling, file inputs, security properties or malformed-document policy.
The handoff should therefore end with a concrete statement such as: In the supplied Python 3.12.14/scikit-learn 1.8.0 reference run, the word-mode custom tokenizer and separately fitted complete analyzer reproduced the authored tokens, named features and counts for D1/D2 and the held-out Q1 transform; the split-only callable analyzer did not. Consumer compatibility remains a separate review. That is reviewable without pretending the fixture proves more than it does.
Build the foundations for maintaining text-feature code
The opening question now has a precise answer. With CountVectorizer, the selected customization point determines which ordinary text rules are eligible to execute. A custom tokenizer in word mode replaces splitting while leaving the surrounding eligible word-analysis stages in the path. A callable analyzer supplies the feature sequence itself, so ordinary preprocessing, stop-word and n-gram settings tested here do not wrap that callable; decoding still precedes it. The authoritative way to accept the implementation is not to read the constructor alone, but to compare observed tokens, names, counts, warnings and controls with an independently authored policy.
For engineers building the broader skills behind that review discipline, Refonte Learning's Data Science & AI programme advertises competencies including Python Data Science, Statistical Modelling, Model Optimization & Problem-Solving Techniques, EDA and Data Visualization, Machine Learning and Predictive Modelling, and application to industry projects. Its FAQ lists Python, Jupyter Notebook, Pandas, NumPy, Matplotlib, scikit-learn and TensorFlow among the software used. Those are relevant foundations for writing and reviewing reproducible feature-engineering experiments. The programme page does not establish that this CountVectorizer laboratory, these exact versions or this specific validation workflow are taught, so the useful next step is simply to strengthen the Python, scikit-learn and problem-solving foundations that make this kind of maintenance evidence possible.