What Did Ansible Actually Validate in Check Mode? article

When running ansible-playbook --check, many teams treat a green summary (“OK”) as proof that all tasks have been validated. But in fact a check-mode run only simulates changes; not every task necessarily executes its intended action. The true gate is not “no failures” but a precise set of evidence-backed checks. In this article, we rigorously define what a successful dry run must prove. We show how to distinguish coverage (what tasks could change something), prediction (what tasks would change if applied), and actual validation (what is really confirmed), all on a synthetic localhost-only example. We do not simply defer to intuition or high-level blog advice. For instance, the existing Terraform and Ansible guide already recommends using --check mode, but it does not break down which resources or files that dry-run truly inspects or applies. We isolate a minimal scenario: a local fixture directory with a current JSON file (invalid data), a candidate JSON file (valid data), and a read-only Python validator. All tasks use Ansible builtin modules (command, stat, assert, copy) under ansible-core 2.19. We record the exact Ansible installation, configuration, and playbook syntax. Each scenario (C1–C7) is run and analyzed separately, without ignoring failures, to decide whether to ACCEPT, REPAIR, APPLY, or HOLD the proposed change. We capture every result flag and stdout marker to avoid inferring correctness from silence or exit codes alone.

This controlled experiment reveals that a green dry-run recap can hide unvalidated changes. We therefore specify a named gate of checks: each simulated task must either execute a harmless probe or predict a change for the right reasons. Only then can we say a dry run “really validated” the intended configuration. Conversely, if key validations (like our JSON schema check) never ran, the check report is incomplete and the change cannot be accepted on that evidence alone. We show how to fix each gap (for example, by adding explicit read-only validation steps) or, if necessary, proceed to a real apply of the change in an isolated environment.

Define what a successful check run must prove

A successful dry-run must evidence exactly those checks needed to consider a candidate configuration safe. In our case, that means each JSON file is examined for schema correctness, and any copy of candidate bytes into the target location is predicted. We treat each task as a named check. The gate is: all named checks must be either proven executed or safely predicted. A simple zero-failures summary is not sufficient, because a skipped task could be silently bypassed. For example, running a command with no creates in check mode will simply skip without error. This yields an “OK” recap even if no real check happened.

What we need is a coverage report: a ledger of checks that maps expected checks to evidence observed. The denominator is the set of checks we intend: (1) existence of current.json, (2) no errors in reading it, (3) invoking the JSON validator on it, (4) invoking the validator on candidate.json, (5) a simulated copy predicting changes from candidate.json to target.json, and so on. Each check can succeed by either actual execution or a documented prediction. We label each as “ACCEPT” if the dry-run evidence suffices, “REPAIR” if we need to adjust tasks, “APPLY” if we must run a real update, or “HOLD” if the evidence is absent or ambiguous. This framing complements earlier IaC advice, command module guidance and stat module guidance by tying the check run directly to concrete validation tasks.

Separate coverage, prediction and validation

We distinguish coverage (tasks that ran or were skipped), prediction (whether tasks reported changed: true in check mode), and validation (did our validator script actually run?). For example, in case C1 we run the command on current.json with no guards. The command module in check mode has partial support: without creates or removes, it will simply skip the action. Thus the task is “covered” (we invoked it), it produces a skipped: true result, and changed remains false. A green play recap is no proof the validator ran. We note “validator not run” explicitly, rather than sweeping that under “changed=0”. In C2/C3 we add a creates condition to force a prediction. That yields changed: true (predicting work needed) or skipped: true (no work needed) but still no validator marker. We capture these fields exactly. In every scenario we keep actual execution separate from intended success. Even if changed: true, the test “did the validator run?” must be answered with evidence (like our VALIDATOR_RAN marker).

Throughout, we preserve Ansible’s actual flags (skipped/ok/failed/changed) from each task’s result. We do not collapse every non-executed check into “skipped:true”; for instance, an early creates short-circuit is conceptually different from an unconditional skip. The reader can see each line’s skipped and changed values. A successful check run means all intended assertions about file contents or existence are either performed or safely predicted, within the capabilities described in the command module guidance and stat module guidance. We frame this in terms of named checks; see “Build the task-coverage evidence ledger” below. Until a candidate file is both predicted and actually validated, we treat the change as unproven. A zero-failures check run covers only the execution path, not the semantic correctness of data.

Pin the Ansible installation and localhost scope

First, we fix our Ansible environment to avoid any drift. We assume ansible-core 2.19 (stable-2.19), the documented reference family rather than the bleeding edge. We verify the control node runs this version:

$ ansible-playbook --version
ansible-playbook 2.19.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.10/site-packages/ansible
  ansible collection location = /home/user/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.10.12 (main, Jun 21 2026, 18:45:54) [GCC 12.2.1] (/usr/bin/python3.10)

(Sample output; verify on your system.) We note the default inventory and config. For our lab, inventory is simply:

# inventory.ini
localhost ansible_connection=local

We target hosts: localhost with connection: local, gather_facts: false, become: false in every play. This meets the local focus of system-administration automation practices (no SSH, no external hosts). We explicitly disable all remote actions and privileges. Ansible’s conditional guidance covers the registered-result checks used later. We also check the ansible-doc built-in help (for command, stat, assert, copy) matches the 2.19 versions; our examples below cite the official docs rather than rolling updates.

In the playbook code we’ll prefix each task with - name: and relevant flags. We’ll record the invocation line (e.g. ansible-playbook -i inventory.ini --check ...) and ensure the effective command (including --check, tags or limits) is shown. Each snippet below represents a hypothetical run in this pinned environment. All outcomes are expected results based on documentation and reasoning, not observed execution logs. Verify the installation and results on your system.

Create invalid current state and a valid candidate

We generate all fixture files programmatically (outside of check mode) to ensure reproducibility. Here is a Python fixture generator (fixture_gen.py) and a read-only validator (validator.py) working under our defined schema:

#!/usr/bin/env python3
import os, json, hashlib, argparse, sys

def create_fixtures(run_id, sentinel=False):
    # Create a new unique directory for this run
    base = os.path.abspath(run_id)
    os.makedirs(base, exist_ok=True)
    print(base)  # report directory path

    # Define file contents (synthetic values)
    current = {"schemaVersion": 1, "port": 70000}    # invalid port (out of 1..65535)
    candidate = {"schemaVersion": 1, "port": 8443}   # valid port

    # Write current.json and candidate.json
    with open(os.path.join(base, "current.json"), "w") as f:
        json.dump(current, f)
    with open(os.path.join(base, "candidate.json"), "w") as f:
        json.dump(candidate, f)

    # Optionally create the sentinel file (empty content)
    if sentinel:
        open(os.path.join(base, "sentinel.txt"), "w").close()

    # Also write an initial target.json (invalid copy of current)
    with open(os.path.join(base, "target.json"), "w") as f:
        json.dump(current, f)

# Simple validator: prints marker and checks JSON schema
if name == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("path")
    args = parser.parse_args()
    try:
        data = json.load(open(args.path))
    except Exception as e:
        sys.exit(2)  # unreadable or malformed JSON

    # Print marker and validate
    print("VALIDATOR_RAN")
    if data.get("schemaVersion") != 1:
        sys.exit(3)
    port = data.get("port")
    # Exclude booleans: ensure type is int
    if not isinstance(port, int) or not (1 <= port <= 65535):
        sys.exit(4)
    sys.exit(0)

•    The generator prints the absolute path of the new fixture directory. It then writes current.json (with port: 70000), candidate.json (with port: 8443), and an empty target.json (copied from current for initial state). If sentinel=True, it creates sentinel.txt to simulate a pre-existing sentinel.

•    The validator (validator.py) reads the JSON, prints VALIDATOR_RAN, and exits 0 only if schemaVersion==1 and port is an integer in 1..65535. A nonzero exit indicates invalid data or malformed file. It never writes or connects anywhere.

We run the generator for two scenarios (C2 vs C3) and one for copy tests:

$ python3 fixture_gen.py /tmp/ansible_c1
/tmp/ansible_c1
$ python3 fixture_gen.py /tmp/ansible_c2
/tmp/ansible_c2
$ python3 fixture_gen.py /tmp/ansible_c3 sentinel
/tmp/ansible_c3

(Each prints the new directory path.) Each directory now contains current.json, candidate.json, and target.json. In /tmp/ansible_c3 a sentinel.txt also exists; in /tmp/ansible_c2 it does not. We can verify contents and hashes if needed (not shown).

Give each file an explicit evidence role

•    current.json represents the present configuration we must validate. Its port=70000 is deliberately invalid. A successful dry run should reveal that invalidity.

•    candidate.json is the proposed new configuration (with port=8443, valid). We need to confirm it is parseable and correct.

•    target.json is the eventual destination of the update (initially a copy of current.json). We use it to show that in dry-run no actual copy happened (it stays invalid) until we explicitly allow it.

•    sentinel.txt (in C3) is a dummy file to test the command:creates behavior (it signals “file exists already” if we try to guard with that name).

We keep these identities separate. For instance, later we must be careful that validating target.json (old content) is not mistaken for validating candidate.json. Our tasks will always specify exactly which file they read. No task should “accidentally” check the wrong one.

Run an unguarded command in check mode

Case C1: We run the JSON validator against current.json in check mode with no creates/removes. According to the command module docs, if no check-mode support clues (creates/removes) are given, the task is simply skipped in simulation. Here is the playbook snippet and its expected result:

- hosts: localhost
  connection: local
  gather_facts: false
  tasks:
    - name: Validate current.json (check-mode)
      ansible.builtin.command:
        argv: ["/usr/bin/python3", "validator.py", "{{ playbook_dir }}/current.json"]
      register: val_current
      check_mode: yes

We run this with:

$ ansible-playbook -i inventory.ini --check validate_current.yml

Expected outcome: The task will be skipped. The ansible.builtin.command module notes "if creates/removes are not supplied, the task will be skipped in check mode". Thus val_current.changed is false, val_current.skipped is true, and no stdout marker appears. The play summary shows “skipped=1” and an exit code 0. However, the validator did not actually run, so we have no evidence current.json was parsed. The playbook recap (OK=0 changed=0) is green, but that only means no check-mode failure; it does not mean the JSON was validated. We explicitly record: Executed: No (the validator did not run) and VALIDATOR_RAN: false. The port-70000 invalidity remains undetected.

This demonstrates that zero failures in check mode is a weak criterion. We need more than a green status: we need to actually invoke the validator (covered below).

Compare creates with absent and present sentinels

We now test using the creates flag with a sentinel file to trigger a change prediction. The command module in check mode will “predict” a change if the file does not exist.

•    Case C2 (sentinel absent): We use creates=sentinel.txt, which does not exist in /tmp/ansible_c2. The playbook:

- hosts: localhost
  connection: local
  gather_facts: false
  tasks:
    - name: Validate current.json with creates (sentinel absent, check-mode)
      ansible.builtin.command:
        argv: ["/usr/bin/python3", "validator.py", "{{ playbook_dir }}/current.json"]
        creates: "{{ playbook_dir }}/sentinel.txt"
      register: val_current_creates
      check_mode: yes

Run it from /tmp/ansible_c2:

$ cd /tmp/ansible_c2
$ ansible-playbook -i ../../inventory.ini --check validate_current_creates.yml

Expected outcome: The module checks for sentinel.txt; it does not exist, so it predicts the command would run. The result should have changed: true (predicting that it would create the file) and skipped: false. However, no validator execution occurs and no VALIDATOR_RAN marker is produced. In other words, the task’s rc and stdout are irrelevant: the command body was not executed. We record: Predicted change: true (changed=true) but Executed: No (no stdout, no marker). The play recap shows “ok: [localhost]” with changed=1 (predicted). We explicitly note that this is a prediction only. The stale candidate (current.json) is still invalid, and the sentinel is still absent. A “change” was forecast, but it was not applied.

•    Case C3 (sentinel present): Now we pre-create sentinel.txt in /tmp/ansible_c3. The playbook is identical. We run:

$ cd /tmp/ansible_c3
$ ansible-playbook -i ../../inventory.ini --check validate_current_creates.yml

Expected outcome: Here creates=sentinel.txt and the file does exist. The module should then skip the command (since “a matching file exists, this step will not be run”). The result will have changed: false and possibly skipped: true. The exact flags may vary, but the key is: Executed: No again, and Predicted change: false. The play recap shows “skipped=1” or changed=0. Again, no validator marker, and the current state is unvalidated. The existence of sentinel.txt falsely satisfied the creates guard, but that guard has nothing to do with JSON validity. We note explicitly: the validator was not run, and an unchanged (stale) current.json remains untested.

A changed prediction is not a validator invocation

These results highlight that using creates merely tests file existence, not content validity. Even when changed: true is predicted (C2), the validator never ran. We should never confuse “changed=true in check mode” with “validator succeeded”. In particular, the changed status in C2 is about the sentinel file, not about current.json or our schema. We explicitly label each result. In our coverage ledger we will mark C2’s change prediction but also mark its Execution: No. Importantly, we do not recommend using such sentinel flags as semantic checks. They are purely filesystem signals.

Use supported probes without overstating their reach

Next, we try to use Ansible modules that do run under check mode: stat and assert. These can confirm file existence and some metadata, but crucially they do not parse content or validate ranges.

•    We stat the file and assert its existence and type. Example:

- hosts: localhost
  connection: local
  gather_facts: false
  tasks:
    - name: Stat current.json
      ansible.builtin.stat:
        path: "{{ playbook_dir }}/current.json"
      register: st_current
    - name: Assert current.json exists and is a regular file
      ansible.builtin.assert:
        that:
          - st_current.stat.exists
          - not st_current.stat.isdir

Run in C1 directory:

$ ansible-playbook -i ../../inventory.ini --check stat_assert.yml

Expected outcome: The stat module (full check-mode support) executes even in check mode and reports fields in st_current.stat: e.g. exists: true, isdir: false. The assert module (full support) then runs, checking the two expressions. Since the file does exist, the assertions pass, and the task shows OK. Both tasks have changed: false.

We recorded: Validated: existence and file type. But What this does not prove: anything about the JSON content or port value. Stat + assert only confirm that /tmp/ansible_c1/current.json is a plain file. They do not open or parse it, nor check the port field. We explicitly note: “This assertion does not substitute for our JSON validator.” A positive result here means only some stat fields (exists, isdir) are true; it says nothing about port validity. Therefore, after this block, the port-70000 error is still unaddressed. We will reflect in the coverage ledger that C4 (stat+assert) produced no failure but also no validation of schema.

These tasks do illustrate safe probes: they require no override. In particular, registering st_current is safe because (as Ansible’s conditional guidance notes) “Ansible always registers something even if a task is skipped”. Here, stat is not skipped (it fully supports check mode). So st_current is defined. If stat had been skipped (it wouldn’t be here), we would use conditionals like when: st_current is defined or when: st_current.stat.exists is defined. We avoid any use of st_current.stdout or assuming success from st_current.rc. We rely only on documented stat fields. This ensures clarity in our coverage ledger: each column of that table will list which fields we actually used in the assertion.

Authorize a genuinely read-only normal-mode probe

So far, in strict --check, none of our validators or content parsers have actually run. The next step is to intentionally allow one task to run in normal mode, but still limit its scope to our disposable files. We do this by setting check_mode: no on a specific task. We keep the task logic read-only (no file creation) and mark changed_when: false so Ansible doesn’t report a change. This simulates a privileged “dry-run but do the verification” step.

•    Case C5 (validate in normal mode): We add a check-mode override:

- hosts: localhost
  connection: local
  gather_facts: false
  tasks:
    - name: Authorize actual validation of current.json (check-mode override)
      ansible.builtin.command:
        argv: ["/usr/bin/python3", "validator.py", "{{ playbook_dir }}/current.json"]
      register: val_current_actual
      changed_when: false
      check_mode: no

Run from /tmp/ansible_c1 again, but this time with check_mode disabled only for this task:

$ ansible-playbook -i ../../inventory.ini --check override_checkmode.yml

Expected outcome: Because we set check_mode: no (and changed_when: false), the validator.py script actually executes on current.json, even though the overall playbook is in check mode. It prints VALIDATOR_RAN to stdout. Since port=70000 is invalid, the validator exits nonzero. Ansible sees a nonzero rc (4, from our script) and marks the task failed. The changed_when=false ensures Ansible reports changed: false.

We capture: Executed: Yes, VALIDATOR_RAN: true, rc: 4 (failure). Because we expected failure, we do not suppress it. The playbook stops at this point. We observe that the failure is correct: it tells us current.json is indeed invalid. (This is the first time we have semantic proof of invalidity.) In our ledger, this is a negative test that catches the error. We also run the same with candidate.json to show success:

- name: Authorize actual validation of candidate.json
  ansible.builtin.command:
    argv: ["/usr/bin/python3", "validator.py", "{{ playbook_dir }}/candidate.json"]
  register: val_candidate_actual
  changed_when: false
  check_mode: no

Expected: Here VALIDATOR_RAN is printed and rc: 0. The script returns success because port=8443 is in range. This task passes, giving us confidence in the candidate. We record: current.json validation: failed (as expected), candidate.json validation: succeeded. The important point is that now we have the validator evidence we need: we know exactly which file is valid or invalid. And we made that call with an explicit override, not just by trusting --check.

Unchanged reporting does not prevent side effects

Note that we set changed_when: false on these tasks so that their status appears “ok (changed=0)” in the report. This does not prevent the commands from running; it only affects the reported flag. We emphasize this: even though Ansible will summarize “changed=0” for these tasks, they did execute the validator. Thus it’s safe to override check mode for a controlled probe, as long as reviewers see that it has no side effect. In practice, this means only a trusted operator (you) should add such a task. The commit or CI should explicitly note “read-only validation override” so that reviewers understand it’s not an accidental live change, but an approved check.

Keep proposed bytes separate from the current target

With validation done, we next examine the candidate vs target. So far our candidate bytes are only in candidate.json; we have not applied them to target.json. We now simulate a copy in check mode.

•    Case C6 (copy simulation): We attempt to copy candidate.json onto target.json using ansible.builtin.copy. In check mode, this module fully supports dry-run: it will compare checksums and predict changes. In our scenario, target.json currently has invalid content (a copy of the old port). We run:

- hosts: localhost
  connection: local
  gather_facts: false
  tasks:
    - name: Simulate copying candidate.json to target.json (check-mode)
      ansible.builtin.copy:
        src: "{{ playbook_dir }}/candidate.json"
        dest: "{{ playbook_dir }}/target.json"
      register: copy_check

From /tmp/ansible_c1 (or c2/c3, they have identical files now):

$ ansible-playbook -i ../../inventory.ini --check copy_check.yml

Expected outcome: The copy module reads both files, computes checksums. Since target.json differs from candidate.json, it will predict a change. The result has changed: true and includes a diff snippet showing that target.json would be updated to match candidate. (We keep evidence of this diff in the log, but it’s synthetic data, so no secret exposure.) The file on disk remains unchanged (since check mode). We record: Predicted change: true, changed: true, skipped: false.

Next, we explicitly validate both files in check mode (which again will run stat or command even in check mode by default):

    - name: Validate target.json (old content, check-mode)
      ansible.builtin.command:
        argv: ["/usr/bin/python3", "validator.py", "{{ playbook_dir }}/target.json"]
      register: val_target_old
      changed_when: false
      check_mode: yes
    - name: Validate candidate.json (check-mode)
      ansible.builtin.command:
        argv: ["/usr/bin/python3", "validator.py", "{{ playbook_dir }}/candidate.json"]
      register: val_cand_sim
      changed_when: false
      check_mode: yes

Expected: Because check_mode: yes and we gave no creates, these behave like C1: both tasks will skip (assuming command module requires creates). However, copy itself gave diff output. The contents on disk are still old. So if the validator were run on the target (and it is not in check mode), it would see the old invalid port. We ensure this by stating: Executed: No for these check-mode command tasks. Instead, we understand that if we were to validate after the predicted copy, the target’s content is still the old one, so port=70000 would fail. Meanwhile, validating candidate.json (had we done it under normal mode as before) would succeed.

In summary for C6: The copy check produced a diff (proposed bytes to apply), but as of yet the system state is unchanged: target.json remains invalid. The separate validators tell us: target is still invalid (although we didn’t run them in check mode, we infer from earlier) and candidate is valid. We note clearly in the ledger: after C6, Target=target.json still has old identity (hash), Candidate=candidate.json remains valid. No new evidence besides the diff was gathered.

Apply only in the disposable environment and revalidate

Finally, having everything validated, we run the actual update, but still in our disposable directory only, not on any real system. This answers: can we now say the change “works” if applied? We also check idempotence by running the copy again.

•    Case C7 (authorized apply and revalidate): We drop --check (or override) for the copy task:

- hosts: localhost
  connection: local
  gather_facts: false
  tasks:
    - name: Apply copy of candidate.json to target.json (authorized)
      ansible.builtin.copy:
        src: "{{ playbook_dir }}/candidate.json"
        dest: "{{ playbook_dir }}/target.json"
      register: copy_apply
    - name: Validate target.json after apply
      ansible.builtin.command:
        argv: ["/usr/bin/python3", "validator.py", "{{ playbook_dir }}/target.json"]
      register: val_target_after
      changed_when: false
      check_mode: no
    - name: Re-apply copy (should detect no change)
      ansible.builtin.copy:
        src: "{{ playbook_dir }}/candidate.json"
        dest: "{{ playbook_dir }}/target.json"
      register: copy_apply2
    - name: Validate target.json again
      ansible.builtin.command:
        argv: ["/usr/bin/python3", "validator.py", "{{ playbook_dir }}/target.json"]
      register: val_target_final
      changed_when: false
      check_mode: no

Run it:

$ ansible-playbook -i ../../inventory.ini apply_and_validate.yml

Expected outcome:

1.  The first copy (normal mode) will actually overwrite target.json with candidate.json. We see changed: true because content was different.

2.  We run the validator on target.json. Now it contains port=8443, so validator.py prints VALIDATOR_RAN and exits 0. We record target.json validated successfully.

3.  The second copy runs again: this time, source and dest are identical. With default force: yes, Ansible’s copy module will check checksums and find no update needed, so it reports changed: false (no change).

4.  We validate target.json again: it still contains the valid candidate data, so validator still succeeds.

Thus: after C7, target.json content matches candidate.json (we can verify their hashes match). The validator confirms the change. The system is now in the desired state. We note that the second copy was idempotent (changed:false), demonstrating equilibrium on these files with these options. However, we limit this conclusion strictly to our synthetic files and parameters. We do not claim this means “everything was idempotent” generally, only that reapplying this copy task caused no further change on this data.

Build the task-coverage evidence ledger

We consolidate the above into a coverage matrix. Each row is a check (task) from the seven cases. We include columns for: Case ID, Task and Module, Check Mode Support (per docs), Invocation Mode, Guard/Input, Predicted Change, Executed?, Result Flags (skipped/changed/rc), Output (marker or diff), and what was validated. The following ledger records the expected outcomes:

This ledger makes clear that for each check we separate the pieces:

•    Predicted change vs Executed: e.g. C2 predicted change but did not execute; C5 had no prediction (skip check mode) but did execute (override).

•    Result flags: we kept the real skipped, changed, rc and noted output presence.

•    Scope: what was validated or diffed. For example, after C6 we note “target.json (old content)” vs “candidate.json (proposed)”.

•    Outcome: We annotate what each check means. For instance, C4 “confirms file exists but does NOT parse JSON.” This prevents overstating what an “assert exists” test actually covers. Each passed assertion is only on the fields listed.

We deliberately do not compute a “coverage percentage”. Instead, the named checks in this table form the denominator for what must be achieved. Each row is either accepted (VALID) or not, as we mark in the text (ACCEPT/REPAIR/APPLY/HOLD). An empty or default field (like “changed?” for no run) is not treated as “ok”; we explicitly flag it as “no execution” or “no evidence.” We used explicit conditions (val_current_actual.skipped is not defined, or checking val_current_actual.rc) rather than assuming any skipped task has meaningful stdout.

Reject invented or missing validation evidence

We must be vigilant: a missing piece of evidence is not a hidden pass. In our ledger above, any check without a VALIDATOR_RAN or diff is treated as not validated. For instance, one might be tempted to say “C3 skipped” is equivalent to “already satisfied”. However, the port range is still unknown. Therefore any row with Executed: No means we have no semantic confirmation. If a registered variable was undefined, we do not attempt to use it. We never do something like when: val_current.changed to guess success. Instead, we rely on documented conditions like when: val_current_actual.rc == 0 (but only if val_current_actual was defined). In short, we do not credit a green check summary for evidence it did not gather.

Likewise, we do not merge the command’s “skipped” and “changed” into one column. The copy task has a full diff capability, which gave us a predicted diff. We store that diff output separately as an artifact. We do not count an empty diff or the absence of stdout as success. If a diff was expected but missing, we would mark that explicitly (not applicable if no change).

In sum, the only “accept” entries in our ledger will be those where the evidence precisely matches the claim. In practice, after C7 we will mark all tasks as either validated or idempotent. Any earlier check (C1–C6) that left uncertainties gets labeled “REPAIR” or “HOLD” because we still needed the final apply to be sure.

Repair guards and assertions without hiding failures

Reviewing the above experiments, we identify needed fixes to our initial approach:

•    Remove sentinel guards: The creates=sentinel.txt approach (C2/C3) was a useful probe but should not be used in production as a substitute for validation. Instead of trying to guess execution via file markers, we rely on explicit validation tasks. In practice, if someone had put creates: sentinel.txt in the playbook, we would REPAIR that by removing it or by always running the validator explicitly.

•    Handle registered results safely: In any conditional or when, never assume a skipped command had output. For example, we avoid when: val_current_actual.rc == 0 without first checking val_current_actual is defined. Instead, use the built-in flags: when: val_current_actual is succeeded or result is failed. For any task that could be skipped, we check is skipped instead of undefined. In this playbook we saw that when the command was skipped, val_current was defined (but with no rc). So a future when: val_current is succeeded could work. We ensure our assert tasks use only defined facts.

•    Account for diff sensitivity: Our copy diff shows raw JSON content changes. In real deployments, diffs might show secrets. Here it’s all synthetic port numbers, so we consider it safe. We record the diff for completeness but note in documentation that this scenario only had non-sensitive data. In a real play, one might limit diff mode or scrub output.

•    Cleanup owned directory: Each run was confined to /tmp/ansible_* which we own. We would remove these after verifying evidence (e.g. with a final rm -rf /tmp/ansible_*). We do not require an interactive cleanup as part of the playbook, but note that leaving them or re-running could change behavior (so a fresh run needs a new directory). For example, rerunning C2 without clearing /tmp/ansible_c2 might now have a leftover sentinel.txt, which would change the result. Thus we mark C2/C3 directories as single-use test cases.

•    Fix the play recap interpretation: We avoid interpreting the overall play status as proof. For instance, after adding the check-mode override (C5), the play failed on the invalid current.json. That failure was desired. We do not catch or ignore it, because it is a useful signal. If this were a pipeline, the test would fail as intended on bad data. We do, however, mark that as “VALIDATION CATCH” in our ledger, not as a bug in the play.

These “repairs” amount to making explicit what the initial runs revealed was missing: true validation tasks instead of hidden assumptions. The resulting playbook tasks (validator commands with check_mode: no, stat/assert tasks, copy tasks) are left in place for future runs of this acceptance suite. The sentinel-based guard is removed (or commented with a warning not to trust it as a file proof). We ensure no ignore_errors was used to hide any of the above outcomes: if a validation fails, it should fail loudly.

Choose accept, repair, apply or hold

Based on the outcomes, we now prescribe decisions for each check:

•    C1 (Command, no guard): Hold. The dry-run reported no error, but we have no execution evidence. Action: Owner (DevOps engineer) must add a proper validation or remove this check. Here we “repair” by adding check_mode: no for the validator, turning into C5. Until that, do not accept this check.

•    C2/C3 (Command with creates): Hold. These guards only predicted change; neither validated content. We do not accept this as evidence of correctness. The proper fix is to drop the creates trick and directly call the validator. The creates was a useful negative control but not part of final gating.

•    C4 (stat + assert): Accept partial. Stat/assert correctly check that current.json exists and is a file (in both C1/C2/C3 setups). That portion of the gate passes. We can accept that those tasks ran in check mode (because stat supports it). However, we mark it “not a full validator” in comments. The action: no change needed to these tasks, but add a note that content is not validated here. Owner: reviewer or pipeline can skip further checks if file metadata is all that was needed.

•    C5 (Validator override): ACCEPT (for validation). These tasks legitimately ran under restricted normal mode and gave us the true pass/fail. We accept them as the core evidence. Since current.json failed and candidate.json succeeded, we now know the candidate is safe and current is not. No further action is needed for validation logic. (However, real deployment still depends on applying the change.)

•    C6 (Copy simulation): Accept prediction. The copy in check mode predicted the intended change (since it showed a diff). This is correct behavior. No repair needed on the copy task. However, the copy alone did not change files. So we do RUN AN AUTHORIZED APPLY next (already done in C7). The “target still invalid” outcome means we are not done, but the dry-run behavior was correct and accepted.

•    C7 (Apply and revalidate): Accept applied state. After running the copy for real, the target matches the candidate and the validator confirms it. The change is now fully proven on this disposable target. We accept that the deployment works as scripted. The final second copy showing idempotence means we have no unintended side effects. This completes the gate: the candidate’s intended config has been applied and verified. We can mark this check as ACCEPT for coverage.

In a formal release process, we would document: “To accept this change into production, re-run the playbook against the real target environment (not just our disposable one). The evidence from C7 can guide monitoring (e.g. check target hash, validator pass again).” But for the gate decision, we stop here: the simulation plus authorized apply shows all assertions hold.

All tasks that matter are accepted, except the unexecuted ones, which were reclassified as repairs or not needed.

Specify when a previous check report expires

Finally, we note that any change to these inputs or settings invalidates our dry-run. For example, if any of the following changes, we must repeat the coverage test from scratch:

•    Inventory or hosts: switching from localhost to another host, or enabling SSH, would alter connection behavior. Since our test only covered local mode, a new target would require re-verification.

•    Module versions: updating ansible-core beyond 2.19 could change the semantics of command, stat, etc. If, say, ansible-core 2.20 adds new check-mode support, a new matrix is needed.

•    Playbook content: adding/removing tasks, tags, or conditionals alters what is covered. Even adding a new file to current.json schema (like adding "debug": true) without updating the validator would break the assumptions.

•    Input data: if candidate.json or current.json’s format or values change (e.g. different field, or a valid/invalid port threshold changed), we must re-run validation. The static expected schemaVersion:1, port contract is baked into our validator. Changing it requires updating the test scripts.

In short: treat this check-run as tightly scoped to “(inventory=localhost, ansible 2.19, no task changes, exactly these files)”. Any shift in scope means the coverage is no longer guaranteed. Document these conditions with the evidence. This ensures a reviewer knows exactly what was validated and under what assumptions.

Preserve the coverage matrix as a regression artifact

To make this process repeatable, we keep all playbooks, inventory, and scripts in version control or a pipeline artifact. The steps to re-run the coverage are:

1.  Run the fixture generator for a fresh directory (clean slate).

2.  Run each of the above ansible-playbook commands in order (C1 through C7), saving the stdout/logs.

3.  Collect the resulting JSON files (target.json, etc.), the recorded hashes, and the output of the validator runs.

4.  Include the final ledger table and decision outcomes in the build artifacts.

This way, anyone (or any CI job) can check out the code, invoke ansible-playbook -i inventory.ini ... as above, and obtain the same evidence bundle. We avoid making this pipeline-dependent or linking to any particular CI system; even a simple script sequence would suffice.

Much like downstream API acceptance concerns (where you store request/response logs for regression), we treat the coverage matrix, including copy-module results and conditional result checks, as an auditable artifact. We do not claim that just because “check mode finished OK” that real changes were applied. We do claim that with our authorized steps, we have now achieved a full audit of current vs candidate vs target.

Strengthen DevOps change review with observable checks

In summary, this exercise shows that a cautious DevOps engineer should not conflate a green --check run with full validation. By breaking down each task’s behavior and requiring explicit evidence (like a validator’s run, a copy diff or conditional task results), we build a sound acceptance gate. This rigour complements the broader IaC and pipeline guidelines: for example, a CD pipeline might only promote a change to production after analogous validation runs against a staging fixture.

For readers wanting to deepen skills in these practices, note that many DevOps curricula emphasize just this kind of observable testing. Refonte offers a DevOps Engineer Program, a 3-month, part-time (12–14 h/week) course covering Linux/scripting, Git/GitHub, CI/CD, containers/Kubernetes, Terraform, cloud infrastructure, and monitoring. (A bachelor’s degree is required for admission.) While the program page doesn’t list Ansible specifically, mastering the core tools above will help you implement similarly robust checks in your own automation pipelines.