Stop Starting systemd Dependents Before the Service Is Ready

A dependent systemd service can start at exactly the wrong moment while every supervisory signal appears superficially reasonable. The producer has a PID. systemctl start may have returned successfully. The producer unit may even be active. Yet the first operation the consumer needs can still fail because application initialization has not reached the state that makes that operation useful.

That is the narrow problem this acceptance playbook tests.

The systemd v257 service documentation distinguishes the startup-completion boundaries for Type=simple, Type=exec, and Type=notify; the unit documentation separately distinguishes ordering relationships from requirement dependencies. Neither distinction by itself proves that an application-specific operation works. The acceptance evidence therefore has to join systemd state, process identity, application milestones, notification, and an independent consumer result. The systemd.service manual source pinned to v257 is the documentation baseline here, not a claim that v257 is the newest systemd release.

This is a disposable, single-host laboratory for Linux administrators, application owners, and platform engineers. It does not modify production units or boot targets. It does not cover socket activation, watchdogs, reload protocols, container termination, Kubernetes, fleet management, or boot-speed tuning.

Execution status: the fixture below is reproducible but was not executed on a disposable systemd VM within this research environment. No command output or pass result is represented as observed. Tables describing experiment outcomes are explicitly expected results derived from the documented semantics and fixture code; an authorized operator must run the fixture and attach measured evidence before approval.

Define the operation that makes the producer ready

Start by refusing to define readiness as “the process exists.” That is useful supervisor information, but it is not the consumer contract.

For this laboratory, the producer owns one local Unix-domain-socket operation:

request:  GET\n

before initialization:
ERR NOT_READY phase=INITIALIZING

after initialization:
OK version=v1

Initialization has one controlled predicate. The producer begins in INITIALIZING, exposes its socket if it has reached that part of execution, and waits for a root-controlled release file. Once the release exists, it reads /etc/lab-readiness/producer.conf. Only a valid version=v1 configuration moves the application to OPERATION_READY. Failed configuration loading exits nonzero; it never announces readiness.

That makes the contract bounded and reviewable:

The application owner owns the claim that “configuration version v1 is initialized sufficiently for this operation.” Linux operations owns how that transition is represented to systemd. The consumer owner owns the independent operation that proves usefulness.

That division matters even where teams already have automation and configuration-management foundations. Installing a file successfully or reporting an automation task as changed/successful is a different assertion from proving that the running producer can serve the consumer’s operation. Refonte’s automation article discusses desired-state and configuration-management concerns; this laboratory deliberately stays one layer lower, at the process/startup boundary.

Freeze the unit files and the host baseline

The documentation baseline is systemd v257, but the VM must record what it actually runs. As of the September 23, 2026 research cutoff, Debian identifies Debian 13.7 as the current Trixie point release, released September 12, 2026; Debian’s stable package pages list systemd, libsystemd0, and libsystemd-dev as 257.13-1~deb13u1. Those repository values are useful for selecting a disposable reference VM, but they are not a substitute for recording installed packages on the machine that generates the acceptance evidence.

A Debian 13.7 VM is therefore a convenient reference target, while the v257 manuals remain the pinned semantic baseline. Vendor patch levels or backports must stay visible.

Install the build prerequisites on the disposable VM:

sudo apt-get update
sudo apt-get install -y build-essential pkg-config libsystemd-dev python3

Before creating any laboratory unit, capture:

mkdir -p "$HOME/lab-readiness-evidence"

{
    echo "=== os-release ==="
    cat /etc/os-release

    echo "=== systemd binary ==="
    systemd --version

    echo "=== installed packages ==="
    dpkg-query -W -f='${Package} ${Version}\n' \
        systemd libsystemd0 libsystemd-dev

    echo "=== pkg-config libsystemd ==="
    pkg-config --modversion libsystemd

    echo "=== boot id ==="
    cat /proc/sys/kernel/random/boot_id
} | tee "$HOME/lab-readiness-evidence/host-baseline.txt"

The local five-second producer startup timeout used later is a test parameter, not a universal recommendation. Restart=no is intentional so that automatic retries do not erase the first failure boundary.

This exercise builds on ordinary Linux administration skills and operating context, but its acceptance question is much narrower than general Linux service management.

Inspect the effective configuration before testing

An important production habit is distinguishing three revisions: what is on disk, what PID 1 has loaded, and what executable is actually running.

The v257 systemctl documentation says systemctl cat prints a unit's backing fragment and drop-ins on disk, and specifically warns that these files may differ from the service manager's understanding if they changed without daemon-reload. systemctl show exposes normalized configuration and runtime state; daemon-reload reloads unit files and recreates the dependency tree. The pinned systemctl v257 manual source documents those boundaries.

For every case, capture:

sudo systemctl cat lab-producer.service lab-consumer.service

sudo systemctl show lab-producer.service \
  -p FragmentPath -p DropInPaths -p Type -p NotifyAccess \
  -p TimeoutStartUSec -p Restart -p ExecStart \
  -p ActiveState -p SubState -p Result -p MainPID -p InvocationID

sudo systemctl show lab-consumer.service \
  -p FragmentPath -p DropInPaths -p Type \
  -p ActiveState -p SubState -p Result \
  -p ExecMainCode -p ExecMainStatus -p InvocationID

sudo sha256sum \
  /etc/systemd/system/lab-producer.service \
  /etc/systemd/system/lab-consumer.service \
  /opt/lab-readiness/bin/lab-producer \
  /opt/lab-readiness/bin/lab-consumer \
  /etc/lab-readiness/producer.conf

After an intentional unit edit, run systemctl daemon-reload; do not confuse it with reloading application configuration. If a process is already running, also record its MainPID, /proc/PID/exe, and executable hash. A changed source file or unit file does not prove that the current process embodies that revision.

Build a producer whose initialization can be held

Create a dedicated identity and directories. The root-owned control directory contains controller input; the service-owned state directory contains producer and consumer evidence.

sudo useradd --system --no-create-home \
  --home-dir /nonexistent --shell /usr/sbin/nologin labready \
  2>/dev/null || true

sudo install -d -o root -g root -m 0755 \
  /opt/lab-readiness/bin \
  /etc/lab-readiness \
  /run/lab-readiness \
  /run/lab-readiness/control

sudo install -d -o labready -g labready -m 0750 \
  /run/lab-readiness/state

printf 'version=v1\n' |
  sudo tee /etc/lab-readiness/producer.conf >/dev/null
sudo chmod 0644 /etc/lab-readiness/producer.conf

The producer below is the single implementation used throughout the comparisons. Its useful workload does not change between simple, exec, and correct notify: it exposes the same socket, waits on the same gate, loads the same configuration, and serves the same GET operation. Negative controls alter only the notification mode so that they can deliberately falsify the startup contract.

Save as lab-producer.c:

#define GNUSOURCE
#include <systemd/sd-daemon.h>
#include <sys/socket.h>
#include <sys/un.h>
#include <sys/stat.h>
#include <poll.h>
#include <signal.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
#include <errno.h>

static volatile sig_atomic_t stopping;

static void on_signal(int sig) { (void)sig; stopping = 1; }

static uint64_t mono_us(void) {
    struct timespec ts;
    if (clock_gettime(CLOCK_MONOTONIC, &ts) != 0) return 0;
    return (uint64_t)ts.tv_sec 1000000ULL + ts.tv_nsec / 1000ULL;
}

static const char run_id(void) {
    const char v = getenv("LAB_RUN_ID");
    return (v && v) ? v : "unset";
}

static void event(const char name, const char detail) {
    printf("run=%s event=%s mono_us=%llu pid=%ld %s\n",
           run_id(), name, (unsigned long long)mono_us(),
           (long)getpid(), detail ? detail : "");
    fflush(stdout);
}

static int phase(const char dir, const char value) {
    char path[512], tmp[512];
    if (snprintf(path, sizeof(path), "%s/phase", dir) >= (int)sizeof(path) ||
        snprintf(tmp, sizeof(tmp), "%s/phase.tmp.%ld",
                 dir, (long)getpid()) >= (int)sizeof(tmp))
        return -1;

    FILE f = fopen(tmp, "w");
    if (!f) return -1;
    if (fprintf(f, "%s\n", value) < 0 ||
        fflush(f) != 0 || fsync(fileno(f)) != 0) {
        fclose(f); unlink(tmp); return -1;
    }
    if (fclose(f) != 0) { unlink(tmp); return -1; }
    if (rename(tmp, path) != 0) { unlink(tmp); return -1; }
    return 0;
}

static int load_version(const char path, char out, size_t n) {
    char line[256];
    FILE f = fopen(path, "r");
    if (!f) return -1;
    if (!fgets(line, sizeof(line), f)) { fclose(f); return -1; }
    if (fclose(f) != 0) return -1;

    line[strcspn(line, "\r\n")] = '\0';
    if (strncmp(line, "version=", 8) != 0 || line[8] == '\0')
        return -1;
    return snprintf(out, n, "%s", line + 8) < (int)n ? 0 : -1;
}

/* 0: no notification channel; 1: enqueued; -1: API/send error. /
static int send_ready(const char sender) {
    const char socket = getenv("NOTIFY_SOCKET");
    char detail[192];

    if (!socket || !socket) {
        snprintf(detail, sizeof(detail),
                 "sender=%s result=no-notify-socket", sender);
        event("READY_NOTIFY", detail);
        return 0;
    }

    int r = sd_notify(0, "READY=1\nSTATUS=operation-ready");
    if (r < 0) {
        snprintf(detail, sizeof(detail),
                 "sender=%s result=error errno=%d text=%s",
                 sender, -r, strerror(-r));
        event("READY_NOTIFY", detail);
        return -1;
    }
    if (r == 0) {
        snprintf(detail, sizeof(detail),
                 "sender=%s result=not-sent", sender);
        event("READY_NOTIFY", detail);
        return -1;
    }

    snprintf(detail, sizeof(detail),
             "sender=%s result=enqueued", sender);
    event("READY_NOTIFY", detail);
    return 1;
}

static int serve_one(int listener, int ready, const char version) {
    int fd = accept4(listener, NULL, NULL, SOCK_CLOEXEC);
    if (fd < 0) return errno == EINTR ? 0 : -1;

    char request[32] = {0}, reply[256];
    ssize_t n = read(fd, request, sizeof(request) - 1);

    if (n <= 0 || strcmp(request, "GET\n") != 0)
        snprintf(reply, sizeof(reply), "ERR BAD_REQUEST\n");
    else if (!ready)
        snprintf(reply, sizeof(reply),
                 "ERR NOT_READY phase=INITIALIZING\n");
    else
        snprintf(reply, sizeof(reply),
                 "OK version=%s\n", version);

    size_t len = strlen(reply), off = 0;
    while (off < len) {
        ssize_t w = write(fd, reply + off, len - off);
        if (w < 0) {
            if (errno == EINTR) continue;
            close(fd);
            return -1;
        }
        off += (size_t)w;
    }

    close(fd);
    event(ready ? "SERVE_OK" : "SERVE_NOT_READY", reply);
    return 0;
}

int main(int argc, char *argv) {
    if (argc != 5) {
        fprintf(stderr, "usage: %s SOCKET CONFIG GATE STATE_DIR\n",
                argv[0]);
        return 64;
    }

    const char socket_path = argv[1];
    const char config_path = argv[2];
    const char gate_path = argv[3];
    const char state_dir = argv[4];
    const char mode = getenv("LAB_NOTIFY_MODE");

    if (!mode || !mode) mode = "correct";
    if (strcmp(mode, "correct") && strcmp(mode, "premature") &&
        strcmp(mode, "never") && strcmp(mode, "child")) {
        fprintf(stderr, "invalid LAB_NOTIFY_MODE=%s\n", mode);
        return 65;
    }

    signal(SIGTERM, on_signal);
    signal(SIGINT, on_signal);
    signal(SIGPIPE, SIG_IGN);

    int listener = socket(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0);
    if (listener < 0) { perror("socket"); return 70; }

    struct sockaddr_un address = { .sun_family = AF_UNIX };
    if (strlen(socket_path) >= sizeof(address.sun_path)) return 71;
    strcpy(address.sun_path, socket_path);

    unlink(socket_path);
    if (bind(listener, (struct sockaddr *)&address, sizeof(address)) ||
        chmod(socket_path, 0660) || listen(listener, 8)) {
        perror("bind/chmod/listen");
        return 72;
    }

    if (phase(state_dir, "INITIALIZING")) return 73;
    char detail[128];
    snprintf(detail, sizeof(detail), "mode=%s", mode);
    event("INITIALIZING", detail);

    if (!strcmp(mode, "premature") &&
        send_ready("main-premature") < 0)
        return 74;

    int ready = 0;
    char version[128] = "unset";

    while (!stopping) {
        if (!ready && access(gate_path, F_OK) == 0) {
            event("GATE_RELEASED", "controller-file-present");

            if (load_version(config_path, version, sizeof(version))) {
                event("INIT_FAILED", "invalid-or-unreadable-config");
                phase(state_dir, "INIT_FAILED");
                return 75;
            }

            ready = 1;
            if (phase(state_dir, "OPERATION_READY")) return 76;

            snprintf(detail, sizeof(detail), "version=%s", version);
            event("OPERATION_READY", detail);

            if (!strcmp(mode, "correct") &&
                send_ready("main") < 0)
                return 77;

            if (!strcmp(mode, "child")) {
                pid_t child = fork();
                if (child < 0) return 78;
                if (child == 0)
                    exit(sendready("child") < 0 ? 79 : 0);

                snprintf(detail, sizeof(detail),
                         "sender_pid=%ld", (long)child);
                event("CHILD_NOTIFY_STARTED", detail);
            }
        }

        struct pollfd pfd = { .fd = listener, .events = POLLIN };
        int r = poll(&pfd, 1, 100);
        if (r < 0) {
            if (errno == EINTR) continue;
            return 80;
        }
        if (r > 0 && (pfd.revents & POLLIN) &&
            serve_one(listener, ready, version))
            return 81;
    }

    event("STOPPING", "signal");
    close(listener);
    unlink(socket_path);
    return 0;
}

Build the same binary used for all service-type comparisons:

cc -std=c17 -O2 -Wall -Wextra -Wpedantic \
  "$(pkg-config --cflags libsystemd)" \
  -o lab-producer lab-producer.c \
  "$(pkg-config --libs libsystemd)"

sudo install -o root -g root -m 0755 \
  lab-producer /opt/lab-readiness/bin/lab-producer

The polling interval merely lets the foreground process notice the explicit release barrier and incoming test requests. Elapsed time is not the readiness predicate.

Build a consumer that fails loudly when started early

The consumer makes exactly one bounded request. No retry loop is allowed because a retry could hide the ordering defect by turning “started too early” into “eventually succeeded.”

Save this complete file as lab-consumer:

#!/usr/bin/env python3
import json
import os
import socket
import sys
import time

if len(sys.argv) != 4:
    print(f"usage: {sys.argv[0]} SOCKET EXPECTED_VERSION RESULT_FILE",
          file=sys.stderr)
    sys.exit(64)

socket_path, expected, result_file = sys.argv[1:]
record = {
    "run_id": os.environ.get("LAB_RUN_ID", "unset"),
    "pid": os.getpid(),
    "request_mono_us": time.monotonic_ns() // 1000,
    "response": None,
    "error": None,
    "ok": False,
}

try:
    with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as s:
        s.settimeout(1.0)
        s.connect(socket_path)
        s.sendall(b"GET\n")
        data = s.recv(256)

    record["response"] = data.decode(
        "utf-8", errors="replace").strip()
    record["ok"] = record["response"] == f"OK version={expected}"
except (OSError, UnicodeError) as exc:
    record["error"] = f"{type(exc).__name__}: {exc}"

record["finish_mono_us"] = time.monotonic_ns() // 1000
print(json.dumps(record, sort_keys=True), flush=True)

tmp = f"{result_file}.tmp.{os.getpid()}"
with open(tmp, "w", encoding="utf-8") as f:
    json.dump(record, f, sort_keys=True)
    f.write("\n")
    f.flush()
    os.fsync(f.fileno())
os.replace(tmp, result_file)

sys.exit(0 if record["ok"] else 30)

Install it:

sudo install -o root -g root -m 0755 \
  lab-consumer /opt/lab-readiness/bin/lab-consumer

Now create the producer unit:

# /etc/systemd/system/lab-producer.service
[Unit]
Description=Readiness acceptance laboratory producer

[Service]
Type=simple
User=labready
Group=labready
Environment=LAB_NOTIFY_MODE=correct
EnvironmentFile=-/run/lab-readiness/control/env
ExecStart=/opt/lab-readiness/bin/lab-producer \
    /run/lab-readiness/state/producer.sock \
    /etc/lab-readiness/producer.conf \
    /run/lab-readiness/control/release \
    /run/lab-readiness/state
Restart=no
TimeoutStartSec=5s
TimeoutStartFailureMode=terminate
TimeoutStopSec=2s

Create the dependent oneshot:

# /etc/systemd/system/lab-consumer.service
[Unit]
Description=Readiness acceptance laboratory consumer
Requires=lab-producer.service
After=lab-producer.service

[Service]
Type=oneshot
User=labready
Group=labready
EnvironmentFile=-/run/lab-readiness/control/env
ExecStart=/opt/lab-readiness/bin/lab-consumer \
    /run/lab-readiness/state/producer.sock \
    v1 \
    /run/lab-readiness/state/consumer.json
Restart=no
TimeoutStartSec=3s

Neither unit has an [Install] section, and neither is enabled. The tests start only these lab-prefixed units explicitly.

Keep activation separate from ordering

Requires= and After= solve different problems. The v257 unit manual says Requires= pulls the listed unit into activation; if that required unit fails to activate and the requiring unit is ordered After= it, the requiring unit is not started. After= itself is an ordering dependency and is explicitly independent of Requires=/Wants=. The systemd.unit v257 manual source documents this separation.

Thus these are not interchangeable:

# Ordering only: does not by itself pull producer into this transaction.
After=lab-producer.service

# Activation relationship only: does not express the desired sequence.
Requires=lab-producer.service

# Laboratory comparison:
Requires=lab-producer.service
After=lab-producer.service

The crucial remaining question is when systemd decides that the producer has finished starting. That answer changes with Type=.

Expose the early-start case with Type=simple

For Type=simple, the v257 service manual says the manager considers the service started immediately after the main process is forked, before the service program has necessarily reached execve(). Follow-up units may therefore proceed before application initialization. It also notes that systemctl start can report success for a simple service even when later executable setup fails.

Create the comparison drop-in:

sudo mkdir -p /etc/systemd/system/lab-producer.service.d

sudo tee /etc/systemd/system/lab-producer.service.d/20-case.conf \
  >/dev/null <<'EOF'
[Service]
Type=simple
NotifyAccess=none
Environment=LAB_NOTIFY_MODE=correct
EOF

sudo systemctl daemon-reload

Reset every case from a stopped state:

sudo systemctl stop lab-consumer.service lab-producer.service \
  2>/dev/null || true
sudo systemctl reset-failed lab-consumer.service lab-producer.service
sudo rm -f /run/lab-readiness/control/release \
  /run/lab-readiness/state/phase \
  /run/lab-readiness/state/consumer.json \
  /run/lab-readiness/state/producer.sock
printf 'LAB_RUN_ID=simple-001\n' |
  sudo tee /run/lab-readiness/control/env >/dev/null

Start the consumer, not the producer. That ensures Requires= genuinely pulls the producer into the transaction:

sudo systemctl start lab-consumer.service
printf 'systemctl_start_consumer_rc=%d\n' "$?"

With the release file absent, the application cannot satisfy its readiness predicate. The reasoned expectation is therefore that the consumer exits 30, with either a connection failure if it beats socket creation or ERR NOT_READY if the producer has already installed the socket. The explicit gate makes successful OK version=v1 impossible before release regardless of VM speed.

Do not call the systemctl return code “the consumer operation result.” They represent different assertions. Record systemctl_start_consumer_rc, ExecMainStatus for lab-consumer.service, and the JSON result separately.

A useful complementary check starts the producer by itself:

sudo systemctl start lab-producer.service
printf 'producer_start_command_rc=%d\n' "$?"

sudo -u labready /opt/lab-readiness/bin/lab-consumer \
  /run/lab-readiness/state/producer.sock v1 \
  /run/lab-readiness/state/manual-consumer.json
printf 'independent_consumer_rc=%d\n' "$?"

Expected, not observed: the producer start job can complete successfully while the independent useful operation still fails. That is precisely the distinction being tested, not an assertion that every active service is unhealthy.

Test why Type=exec is not an application readiness signal

Change only the startup contract:

sudo tee /etc/systemd/system/lab-producer.service.d/20-case.conf \
  >/dev/null <<'EOF'
[Service]
Type=exec
NotifyAccess=none
Environment=LAB_NOTIFY_MODE=correct
EOF

sudo systemctl daemon-reload

The v257 service reference says Type=exec delays the completion of startup until the service executable has been successfully executed. Unlike simple, errors such as a missing executable or invalid service user can therefore make the start command fail. But the same documentation explicitly warns that exec does not propagate failures in the application's own startup code or provide a boundary for completion of application initialization.

Run the same reset and start the consumer with the gate absent.

Expected: systemd knows that execve() of the real producer succeeded, but that says nothing about whether version=v1 has been initialized. The dependent consumer can still run while the producer is held at INITIALIZING, and its one operation must fail.

The comparison is:

The first two rows come directly from documented service semantics. The third is the reason this playbook includes an independent probe rather than treating notification syntax as acceptance.

Distinguish exec failure from initialization failure

First create a controlled missing-executable case:

sudo tee /etc/systemd/system/lab-producer.service.d/30-missing.conf \
  >/dev/null <<'EOF'
[Service]
Type=exec
ExecStart=
ExecStart=/opt/lab-readiness/bin/does-not-exist
EOF

sudo systemctl daemon-reload
sudo systemctl stop lab-consumer.service lab-producer.service \
  2>/dev/null || true
sudo systemctl reset-failed lab-consumer.service lab-producer.service

sudo systemctl start lab-consumer.service
printf 'start_rc=%d\n' "$?"

sudo systemctl show lab-producer.service \
  -p ActiveState -p SubState -p Result \
  -p ExecMainCode -p ExecMainStatus

Expected: executable launch fails, so Type=exec does not reach its start-complete boundary; with Requires= plus After=, the dependent should not be accepted as a successful consumer run.

Restore the legitimate command before proceeding:

sudo rm -f \
  /etc/systemd/system/lab-producer.service.d/30-missing.conf
sudo systemctl daemon-reload

The three observations must stay distinct in the ledger:

1. Missing executable: systemd launch failure.

2. Real executable, gate held: successful exec but application not initialized; consumer operation fails.

3. Process exits during initialization: producer's application failure, recorded separately from whether exec initially succeeded.

For the third case, place malformed content such as bad=v1 in the config, release the gate, record the producer's nonzero exit, then restore version=v1. Never infer “exec failure” simply because a real process subsequently failed during initialization.

Announce READY=1 only after the useful operation is available

Now use the startup contract that can represent the application transition itself:

sudo tee /etc/systemd/system/lab-producer.service.d/20-case.conf \
  >/dev/null <<'EOF'
[Service]
Type=notify
NotifyAccess=main
Environment=LAB_NOTIFY_MODE=correct
EOF

sudo systemctl daemon-reload

For Type=notify, systemd's v257 service documentation says startup remains incomplete until the service sends READY=1; follow-up units proceed after that notification. With NotifyAccess=main, only notifications from the service's main process are accepted.

The producer implements this order:

main process starts
→ INITIALIZING
→ wait for controller release
→ load and validate version=v1
→ OPERATION_READY
→ main process calls sd_notify("READY=1")
→ systemd may complete producer startup
→ After= permits consumer startup
→ consumer issues GET

That is deliberately stronger than “port opened” or “process exists.”

The chosen implementation uses sd_notify() directly from the actual foreground main process. The pinned sd_notify v257 API documentation states that a negative return is an errno-style error, zero means no notification was sent because $NOTIFY_SOCKET was not set, and a positive value means the notification was sent/enqueued. Importantly, that return value does not prove that PID 1 ultimately processed or accepted the message.

That is why the source handles a negative result as an application error, records a missing notification channel separately for the simple/exec comparisons, and still requires systemd plus consumer evidence for the notify case.

Start the consumer with the release absent. Once the producer has written INITIALIZING, inspect:

sudo systemctl show lab-producer.service \
  -p MainPID -p Type -p NotifyAccess \
  -p ActiveState -p SubState -p Result

sudo systemctl list-jobs --no-pager

Then release initialization:

sudo touch /run/lab-readiness/control/release

Expected, not observed: no consumer result exists before the release; after configuration loading and the main-process notification, the consumer runs and receives OK version=v1.

Falsify the notification with a premature-ready case

A correct Type=notify declaration is not enough. The application can assert the wrong transition.

The source already includes a deliberately bad premature mode. It sends READY=1 from the correct main PID while the phase is still INITIALIZING, before the controller releases the gate:

sudo tee /etc/systemd/system/lab-producer.service.d/20-case.conf \
  >/dev/null <<'EOF'
[Service]
Type=notify
NotifyAccess=main
Environment=LAB_NOTIFY_MODE=premature
EOF

sudo systemctl daemon-reload

Reset the fixture, leave /run/lab-readiness/control/release absent, and start lab-consumer.service.

The expected sequence is:

producer main PID starts
producer writes INITIALIZING
producer main PID sends READY=1
systemd completes notify startup
After= boundary opens
consumer performs GET
GET fails because initialization is still held

This is the most important negative control in the laboratory. The sender can be correct. The unit can be syntactically correct. systemd can legitimately honor the application's notification. The dependent operation can still fail because the application asserted readiness too early.

After collecting the negative evidence, create the release file so the producer can finish its local initialization, then stop both units. Do not convert this mode into a fallback strategy.

The supervisor is not independently re-running your application's initialization predicate. READY=1 communicates the application's assertion that startup has finished; systemd's documented role is to use that assertion as the start-completion boundary for Type=notify.

Treat notification as an assertion with an owner

The application owner must therefore name what READY=1 covers.

For this fixture, the claim is deliberately small:

Configuration version v1 has been successfully loaded, and the local GET operation is now able to return OK version=v1.

It does not claim that every downstream system is reachable, every cache is warm, every remote API is healthy, or the process will remain healthy indefinitely.

A different application can legitimately need a different predicate. Opening a listening socket, constructing an in-memory index, loading credentials, reaching a local database, or populating a cache are separate transitions. The acceptance decision is only meaningful when the producer owner says which of those are included in the notification and the dependent owner confirms that the probe tests the operation it actually requires.

Check missing notifications and sender attribution

A Type=notify service that never sends accepted READY=1 must not be allowed to remain indefinitely in this test. The fixture's TimeoutStartSec=5s provides a deliberately short local bound.

The v257 service manual says TimeoutStartSec= sets the time allowed for startup; a daemon that does not signal startup completion within that interval is considered failed and shut down according to the configured failure mode. That is exactly what the missing-notification negative control needs.

Use:

sudo tee /etc/systemd/system/lab-producer.service.d/20-case.conf \
  >/dev/null <<'EOF'
[Service]
Type=notify
NotifyAccess=main
Environment=LAB_NOTIFY_MODE=never
EOF

sudo systemctl daemon-reload

Start the dependent while the gate is held, then release the gate. In this mode the producer reaches OPERATION_READY but intentionally sends no READY=1.

Expected: the useful local operation may be available internally, yet systemd keeps the producer's start job incomplete; after the configured five-second laboratory timeout, producer activation fails and the ordered required consumer must not be accepted as a successful run. Record the actual Result, ActiveState, SubState, and journal records instead of substituting the expected labels for observations.

The second negative control tests sender policy without introducing an unrelated external shell notifier:

sudo tee /etc/systemd/system/lab-producer.service.d/20-case.conf \
  >/dev/null <<'EOF'
[Service]
Type=notify
NotifyAccess=main
Environment=LAB_NOTIFY_MODE=child
EOF

In child mode the real application reaches its operation-ready state, then its main process forks an internal child that calls sd_notify(). The parent logs the child PID. The actual passing implementation is still the main-process notification path; this child path exists only to test attribution.

The v257 service reference states that NotifyAccess=main accepts service status notifications only from the main process. The sd_notify reference also warns that notification attribution has process-lifetime considerations, especially for auxiliary processes that send and disappear.

Before changing policy, capture:

sudo systemctl show lab-producer.service \
  -p MainPID -p Type -p NotifyAccess \
  -p ActiveState -p SubState

sudo journalctl -b -u lab-producer.service \
  -o short-monotonic --no-pager

With NotifyAccess=main, the internally forked child's READY=1 is not the permitted main-process notification. Do not broaden the unit to NotifyAccess=all merely because startup is stuck. If a real production investigation cannot establish which PID sent the notification or whether PID 1 attributed it, the decision is hold, not “change access until it starts.”

Also remember the API distinction: the child seeing a positive sd_notify() return means its message was enqueued, not that the manager accepted that sender's readiness assertion.

Restore LAB_NOTIFY_MODE=correct and NotifyAccess=main for the positive control.

Join manager state, journal events and probe results

The acceptance artifact is an event ledger, not a screenshot of systemctl status.

Use one schema across every run:

The v257 journalctl manual supports unit filtering with -u, current-boot filtering with -b, and short-monotonic output for monotonic timestamps. The journalctl v257 manual source documents short-monotonic; boot selection adds BOOTID matching.

Capture both readable and structured evidence:

sudo journalctl -b \
  -u lab-producer.service \
  -u lab-consumer.service \
  -o short-monotonic --no-pager \
  > "$HOME/lab-readiness-evidence/journal-short.txt"

sudo journalctl -b \
  -u lab-producer.service \
  -u lab-consumer.service \
  -o json --no-pager \
  > "$HOME/lab-readiness-evidence/journal.json"

sudo systemctl show lab-producer.service lab-consumer.service \
  > "$HOME/lab-readiness-evidence/systemctl-show.txt"

cat /run/lab-readiness/state/consumer.json \
  > "$HOME/lab-readiness-evidence/consumer.json" 2>/dev/null || true

The journal is one evidence plane among several. For broader conceptual context, Refonte Learning's article on the roles of metrics, logs and traces discusses observability categories; this acceptance test intentionally uses logs plus explicit process/unit properties and an application operation rather than treating a log line as health proof.

Do not compare monotonic timestamps from unrelated boots as though they shared a clock origin. Retain the boot ID with every run.

Reconcile the first successful consumer operation

For a passing correct-notify run, the measured ledger should establish this ordering:

INITIALIZING
< GATE_RELEASED
< OPERATION_READY
< READY_NOTIFY from main
< systemd startup completion
< consumer request
< consumer OK

The first two application inequalities come from the source code. The systemd relationship between accepted READY=1 and follow-up units comes from documented Type=notify semantics. The final relationship must be established by the actual consumer evidence.

Retain the negative cases too. A complete acceptance package should make the contrast easy to audit: simple and exec allow an early operation; premature notify falsifies the assertion; correct notify defers the consumer until the intended transition.

A producer journal line that says OPERATION_READY is not the independent proof. The consumer response is.

A compact harness can make the release step repeatable without replacing it with “sleep for N seconds.” This script changes only lab-prefixed unit drop-ins:

#!/usr/bin/env bash
set -euo pipefail

case_name="${1:?usage: $0 simple|exec|notify|premature|never|child}"

case "$case_name" in
  simple)    type=simple; access=none; mode=correct ;;
  exec)      type=exec;   access=none; mode=correct ;;
  notify)    type=notify; access=main; mode=correct ;;
  premature) type=notify; access=main; mode=premature ;;
  never)     type=notify; access=main; mode=never ;;
  child)     type=notify; access=main; mode=child ;;
  *) exit 64 ;;
esac

wait_file() {
  local file="$1" limit="$2" deadline=$((SECONDS + limit))
  while [[ ! -e "$file" ]]; do
    (( SECONDS >= deadline )) && return 1
    sleep 0.05
  done
}

wait_phase() {
  local wanted="$1" limit="$2" deadline=$((SECONDS + limit))
  while [[ "$(cat /run/lab-readiness/state/phase 2>/dev/null || true)" \
           != "$wanted" ]]; do
    (( SECONDS >= deadline )) && return 1
    sleep 0.05
  done
}

systemctl stop lab-consumer.service lab-producer.service \
  2>/dev/null || true
systemctl reset-failed lab-consumer.service lab-producer.service
rm -f /run/lab-readiness/control/release \
      /run/lab-readiness/control/start.rc \
      /run/lab-readiness/state/phase \
      /run/lab-readiness/state/consumer.json \
      /run/lab-readiness/state/producer.sock

run_id="${case_name}-$(date -u +%Y%m%dT%H%M%SZ)-$$"
printf 'LAB_RUN_ID=%s\n' "$run_id" \
  > /run/lab-readiness/control/env

cat > /etc/systemd/system/lab-producer.service.d/20-case.conf <<EOF
[Service]
Type=$type
NotifyAccess=$access
Environment=LAB_NOTIFY_MODE=$mode
EOF

systemctl daemon-reload

(
  set +e
  systemctl start lab-consumer.service
  printf '%d\n' "$?" > /run/lab-readiness/control/start.rc
) &

wait_file /run/lab-readiness/state/phase 3

case "$case_name" in
  simple|exec|premature)
    wait_file /run/lab-readiness/state/consumer.json 3 || true
    touch /run/lab-readiness/control/release
    ;;
  notify)
    test ! -e /run/lab-readiness/state/consumer.json
    systemctl list-jobs --no-pager
    touch /run/lab-readiness/control/release
    wait_file /run/lab-readiness/state/consumer.json 3
    ;;
  never|child)
    touch /run/lab-readiness/control/release
    wait_phase OPERATION_READY 3
    wait_file /run/lab-readiness/control/start.rc 8 || true
    ;;
esac

wait || true

systemctl show lab-producer.service lab-consumer.service \
  -p Id -p Type -p NotifyAccess -p MainPID \
  -p ActiveState -p SubState -p Result \
  -p ExecMainCode -p ExecMainStatus -p InvocationID

journalctl -b -u lab-producer.service -u lab-consumer.service \
  -o short-monotonic --no-pager

cat /run/lab-readiness/state/consumer.json 2>/dev/null || true

The 50 ms poll only notices explicit state changes; it does not decide readiness. The five-second and three-second limits remain local test bounds.

Choose a repair that preserves the startup contract

When the evidence shows that a dependent is crossing the boundary too early, the preferred repair is to move the startup-completion assertion to the transition that actually makes the required operation usable.

For an application that can integrate with libsystemd, that means keeping the daemon in the foreground and sending READY=1 from the correct process after the required initialization predicate succeeds. systemd's own v257 guidance notes that notify requires explicit service-code support so application code can determine when startup is complete.

Do not “fix” the race with:

sleep 10

That substitutes elapsed time for state. It can fail on a slow run and waste time on a fast run.

Do not suppress a consumer's failure code. Do not add an unbounded retry loop that makes the same defective ordering look successful eventually. Do not relabel an application Type=notify if no process actually implements the notification protocol.

Software that cannot be modified may need a separate bounded readiness adapter, but that is a new design rather than a synonym for native notification. Such an adapter must own a real application probe, have a finite timeout, propagate failure, have clearly attributable process semantics, and pass the same premature-ready and timeout controls. A unit-only edit cannot manufacture knowledge of an application predicate that no process or probe can observe.

Configuration deployment remains a separate responsibility. Ansible-driven configuration work as a separate layer can distribute units and binaries, but deployment automation should consume this startup contract rather than be mistaken for proof of it.

Approve, repair, hold or revert from the evidence

Acceptance should be a small decision procedure, not a debate over whether systemctl status “looks healthy.”

“Approve” applies only to the tested startup transition on the recorded host configuration. A few controlled runs are useful for catching deterministic contract errors; they are not a statistical availability claim.

A practical local pass criterion is:

1. Start every case from stopped/reset state.

2. Confirm simple fails the independent operation while initialization is held.

3. Confirm exec also fails the same held operation despite successful executable launch.

4. Confirm a nonexistent executable is distinguishable from initialization delay.

5. Confirm correct notify does not execute the consumer before the gate is released.

6. Confirm correct notify produces the expected consumer response after initialization.

7. Confirm premature notify remains a failing negative control.

8. Confirm missing/unauthorized notification reaches a bounded failure rather than waiting forever.

9. Confirm effective units, executable hashes, PIDs, boot ID and journal chronology are retained.

Those are acceptance assertions for this fixture, not universal service-level objectives.

Revert the application and unit as a tested pair

Rollback is also contractual.

Keep the prior application binary, unit fragments, drop-ins, configuration and their hashes together. If a candidate changes the notification behavior, reverting only Type= while leaving a new application binary or stale drop-in in place does not reproduce the previous startup contract.

A laboratory rollback should therefore:

sudo systemctl stop lab-consumer.service lab-producer.service

# Restore the previously recorded binary/unit/configuration pair here.

sudo systemctl daemon-reload
sudo systemctl reset-failed \
  lab-producer.service lab-consumer.service

Then repeat the prior accepted consumer test and collect a fresh ledger. systemctl cat, hashes, systemctl show, and the probe must agree on the restored revision before calling the revert successful. The systemctl documentation's warning that on-disk unit content can differ from the manager's loaded understanding is exactly why daemon-reload and post-restore inspection are required.

Assign startup ownership separately from ongoing health

Startup readiness crosses organizational boundaries, so ownership needs to be explicit.

The application team owns the initialization predicate: which local state must exist before READY=1, which errors prevent it, and whether that state is versioned.

Linux/platform operations owns the unit semantics: Type=, NotifyAccess=, requirement and ordering edges, finite startup timeout, effective configuration inspection, journal capture, and version recording.

The dependent owner owns the useful probe: the exact operation that must work and the response or exit code that constitutes success.

Revalidate the contract after changes to the producer binary, initialization dependencies, notification code, relevant unit/drop-in settings, service identity, configuration format, or consumer requirement. A package update that changes systemd or libsystemd also belongs in the evidence record even when the intended semantics are unchanged.

This separation should not be confused with monitoring and logging tool responsibilities. Monitoring after startup answers a different operational question. This fixture stops at the startup handoff.

A service that becomes ready and fails ten minutes later needs ongoing detection and recovery mechanisms outside this playbook. Passing this test does not establish continuous availability, watchdog correctness, reload safety, or downstream health.

That limit is deliberate. The narrow contract is stronger because every retained artifact answers the same question: when systemd allowed the dependent to start, could the producer already serve the operation the dependent required?

Build system administration practice around observable startup

This laboratory combines several foundations that matter in practical system administration: Linux service configuration, command-line investigation, troubleshooting, process identity, version control of configuration, source compilation, dependency reasoning, and log interpretation.

Refonte Learning's System Administration program currently lists a six-month period and 10–12 hours per week, with competencies including system installation and configuration, network administration, security practices, troubleshooting, virtualization, backup and recovery, command-line work and cloud management. Its page also describes practical projects and guidance. Those verified foundations are relevant preparation for this kind of laboratory; this article does not claim that libsystemd programming, READY=1, or this exact two-service exercise is part of the curriculum.

For a production handover, do not deliver only the statement “we switched to Type=notify.” Deliver the versioned producer binary, the consumer probe, the effective producer and consumer units, every drop-in, the exact host/systemd/libsystemd baseline, the initialization predicate, timeout values, hashes, boot and invocation identifiers, negative-control evidence, the first successful dependent operation, and the approve/repair/hold/revert decision.

That package captures the real startup contract: not merely that Linux launched the producer, but that the dependent was released only at a boundary demonstrated to be useful for the operation it actually needs.