Logrotate Finished, but the Application Is Writing to the Old File

A successful logrotate invocation can leave an application writing somewhere other than the pathname an administrator is watching. The misleading case is simple: app.log exists after rotation, its mode and ownership look correct, logrotate returned zero, yet the file stays empty while new application records appear to be missing. The decisive question is not whether the pathname exists. It is which file object the live process's descriptor references.

On Linux, renaming a file does not invalidate open descriptors referring to it; the Linux rename(2) documentation explicitly states that open file descriptors for the old pathname are unaffected. With conventional rename/create rotation, that means the application can continue appending to the inode now reachable as app.log.1, while logrotate has created a different inode at app.log. The logrotate 3.22.0 maintainer manual documents the rotation and create mechanics, but its successful completion is not an application-descriptor acceptance test.

This playbook builds an isolated acceptance test around that distinction. It uses one unprivileged foreground Python writer, a user-owned temporary directory, controlled sequence IDs, /proc/<pid>/fd, fdinfo, an isolated state file, an explicit reopen contract, and a separate Python WatchedFileHandler comparison. No measured laboratory results are invented here: outputs described as “expected” are predictions derived from the documented mechanisms and must be replaced by captured evidence when the fixture is actually run. The fixed reference set was rechecked on September 24, 2026; logrotate claims are tied to tag 3.22.0 and Python claims to the Python 3.14 documentation rather than to “latest” releases.

Define local log continuity before looking at the scheduler

Define the acceptance contract before running logrotate. Otherwise an empty active file can be misclassified as loss when the records merely landed in the rotated inode.

For this fixture, records have a run ID and monotonically declared sequence ID:

run=rename-sticky seq=000001 fixture=refonte-logrotate-fd-v1

For a six-record run, the expected set is {1,2,3,4,5,6}. “Missing” means an expected sequence is absent from all retained files. “Duplicate” means the same sequence occurs more than once. “Unexpected” means a record belongs to another run, has an undeclared sequence, or violates the complete-line format. “Routing failure” is different: a record exists exactly once but was written to the wrong retained file after rotation.

That distinction matters. With rename/create and a writer that does not reopen, the expected failure is commonly zero missing records but incorrect routing: sequences 1–3 are already in the old inode; after rotation sequences 4–6 continue into that same inode, now named app.log.1. The new app.log can therefore be empty without proving data loss. The mechanism follows directly from the Linux rule that rename leaves existing descriptors unaffected and the logrotate rule that create constructs the replacement pathname after rotation.

An application acknowledgment has a narrower meaning. In this playbook, ACK emit means the fixture completed and flushed that record through its Python stream; ACK reopen means it opened and validated a descriptor for the intended current app.log pathname before replacing its previous descriptor. Neither acknowledgment establishes storage crash durability or remote delivery.

Keep that boundary separate from monitoring and logging stack responsibilities. That Refonte article discusses broader collection and monitoring components; this acceptance gate answers the narrower local question of which inode the writer is using.

The excluded claims are intentional: this test does not prove log shipping, collector ingestion, retention compliance, journald behavior, crash durability, or continuous production losslessness.

Create a disposable writer and private rotation environment

Use a fresh disposable Linux VM and operate entirely as a normal user. Do not substitute /var/log, /etc/logrotate.conf, /etc/logrotate.d, a system state file, or a shared application directory.

The first artifact prepares the directory, verifies the two version constraints, captures the execution environment, and creates the private logrotate configuration.

#!/usr/bin/env bash
# prepare.sh
set -euo pipefail

LAB=${1:?usage: prepare.sh /path/to/user-owned-temp-dir}
PYTHON_BIN=${PYTHON_BIN:-python3.14}
LOGROTATE_BIN=${LOGROTATE_BIN:-logrotate}
FIXTURE_REVISION=refonte-logrotate-fd-v1

mkdir -p "$LAB"
chmod 0700 "$LAB"
LAB=$(cd "$LAB" && pwd -P)

case "$("$LOGROTATE_BIN" --version 2>&1 | head -n1)" in
  3.22.0) ;;
  *) echo "ERROR: this fixture requires installed logrotate 3.22.0" >&2; exit 2 ;;
esac

"$PYTHON_BIN" - <<'PY'
import sys
if sys.version_info[:2] != (3, 14):
    raise SystemExit(f"ERROR: Python 3.14 required, got {sys.version}")
PY

USER_NAME=$(id -un)
GROUP_NAME=$(id -gn)

umask 0027
: > "$LAB/app.log"
chmod 0640 "$LAB/app.log"

cat > "$LAB/logrotate.conf" <<EOF
$LAB/app.log {
    rotate 4
    size 1
    nocompress
    notifempty
    create 0640 $USER_NAME $GROUP_NAME
}
EOF

{
  echo "fixture_revision=$FIXTURE_REVISION"
  echo "captured_utc=$(date -u +%FT%TZ)"
  echo "lab=$LAB"
  echo "user_group=$(id)"
  echo "umask=$(umask)"
  echo
  echo "[os-release]"
  cat /etc/os-release 2>/dev/null || true
  echo
  echo "[kernel]"
  uname -srvmo
  echo
  echo "[logrotate]"
  "$LOGROTATE_BIN" --version
  echo
  echo "[python]"
  "$PYTHON_BIN" --version
  "$PYTHON_BIN" -c 'import sys; print(sys.executable); print(sys.version)'
  echo
  echo "[mount]"
  findmnt -T "$LAB" -o TARGET,SOURCE,FSTYPE,OPTIONS 2>/dev/null || true
  stat -f "$LAB"
  echo
  echo "[package-builds]"
  if command -v dpkg-query >/dev/null; then
      dpkg-query -W logrotate 2>/dev/null || true
      dpkg-query -S "$(command -v "$PYTHON_BIN")" 2>/dev/null || true
  elif command -v rpm >/dev/null; then
      rpm -qf "$(command -v "$LOGROTATE_BIN")" 2>/dev/null || true
      rpm -qf "$(command -v "$PYTHON_BIN")" 2>/dev/null || true
  elif command -v apk >/dev/null; then
      apk info -W "$(command -v "$LOGROTATE_BIN")" 2>/dev/null || true
      apk info -W "$(command -v "$PYTHON_BIN")" 2>/dev/null || true
  fi
  echo
  echo "[config-sha256]"
  sha256sum "$LAB/logrotate.conf"
} > "$LAB/environment.txt"

printf 'Prepared %s\n' "$LAB"

Create the directory outside the script so the shell retains its name:

export LAB="$(mktemp -d "${TMPDIR:-/tmp}/refonte-logrotate.XXXXXX")"
chmod 0700 "$LAB"
./prepare.sh "$LAB"

The alternate state-file option is documented specifically by logrotate, and debug mode is documented not to change logs or update state. Those properties let the test avoid the host-wide state file and distinguish configuration inspection from a real rotation.

The writer below uses an AF_UNIX socket as its private control channel. Record data goes only to app.log; command acknowledgments return over the socket. The custom mode opens a replacement descriptor before closing the old one, so a failed reopen can preserve the prior sink. The watched mode is reserved for the later WatchedFileHandler comparison.

#!/usr/bin/env python3
# writer.py
import argparse
import json
import logging
from logging.handlers import WatchedFileHandler
import os
from pathlib import Path
import socket
import stat

REV = "refonte-logrotate-fd-v1"

def ident(stream):
    st = os.fstat(stream.fileno())
    return {"fd": stream.fileno(), "dev": st.st_dev, "ino": st.st_ino}

def validate_regular(stream, path):
    fst = os.fstat(stream.fileno())
    pst = os.stat(path)
    if not stat.S_ISREG(fst.st_mode):
        raise RuntimeError("target is not a regular file")
    if (fst.st_dev, fst.st_ino) != (pst.st_dev, pst.st_ino):
        raise RuntimeError("opened descriptor no longer matches active pathname")

class CustomSink:
    def init(self, path):
        self.path = path
        self.stream = open(path, "a", encoding="utf-8", buffering=1)
        validate_regular(self.stream, path)

    def emit(self, line):
        self.stream.write(line + "\n")
        self.stream.flush()
        return ident(self.stream)

    def reopen(self):
        old = self.stream
        candidate = open(self.path, "a", encoding="utf-8", buffering=1)
        try:
            validate_regular(candidate, self.path)
        except Exception:
            candidate.close()
            raise
        self.stream = candidate
        old.flush()
        old.close()
        return ident(self.stream)

    def close(self):
        self.stream.close()

class WatchedSink:
    def init(self, path):
        self.handler = WatchedFileHandler(
            path, mode="a", encoding="utf-8", delay=False
        )
        self.handler.setFormatter(logging.Formatter("%(message)s"))
        self.logger = logging.getLogger(f"fixture.{os.getpid()}")
        self.logger.setLevel(logging.INFO)
        self.logger.propagate = False
        self.logger.handlers[:] = [self.handler]

    def emit(self, line):
        self.logger.info(line)
        self.handler.flush()
        return ident(self.handler.stream)

    def reopen(self):
        raise RuntimeError("explicit reopen is not part of watched mode")

    def close(self):
        self.handler.close()

def start_ticks():
    text = Path("/proc/self/stat").read_text()
    fields_after_comm = text.rsplit(")", 1)[1].split()
    return int(fields_after_comm[19])  # field 22 overall

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--log", required=True)
    ap.add_argument("--socket", required=True)
    ap.add_argument("--pidfile", required=True)
    ap.add_argument("--run", required=True)
    ap.add_argument("--mode", choices=("custom", "watched"), default="custom")
    args = ap.parse_args()

    if not args.run.replace("-", "").replace("_", "").replace(".", "").isalnum():
        raise SystemExit("run ID must contain only letters, digits, dot, dash, underscore")

    log_path = str(Path(args.log).resolve())
    sock_path = Path(args.socket)
    pidfile = Path(args.pidfile)

    sink = CustomSink(log_path) if args.mode == "custom" else WatchedSink(log_path)

    if sock_path.exists():
        sock_path.unlink()

    server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
    server.bind(str(sock_path))
    os.chmod(sock_path, 0o600)
    server.listen(1)

    identity = {
        "pid": os.getpid(),
        "starttime_ticks": start_ticks(),
        "boot_id": Path("/proc/sys/kernel/random/boot_id").read_text().strip(),
        "run": args.run,
        "mode": args.mode,
        "fixture": REV,
    }
    pidfile.write_text(json.dumps(identity) + "\n")
    os.chmod(pidfile, 0o600)
    print(json.dumps({"status": "READY", identity}), flush=True)

    stopping = False
    while not stopping:
        conn, = server.accept()
        with conn:
            request = conn.recv(4096).decode("utf-8").strip()
            parts = request.split()
            try:
                if len(parts) == 2 and parts[0] == "emit":
                    seq = int(parts[1])
                    line = f"run={args.run} seq={seq:06d} fixture={REV}"
                    fd = sink.emit(line)
                    reply = {"status": "ACK", "command": "emit",
                             "run": args.run, "seq": seq, fd}
                elif parts == ["reopen"]:
                    fd = sink.reopen()
                    reply = {"status": "ACK", "command": "reopen",
                             "run": args.run, fd}
                elif parts == ["status"]:
                    stream = sink.stream if args.mode == "custom" else sink.handler.stream
                    reply = {"status": "ACK", "command": "status",
                             "run": args.run, ident(stream)}
                elif parts == ["stop"]:
                    reply = {"status": "ACK", "command": "stop", "run": args.run}
                    stopping = True
                else:
                    raise ValueError("expected: emit N | reopen | status | stop")
            except Exception as exc:
                reply = {"status": "ERR",
                         "command": parts[0] if parts else "",
                         "error": f"{type(exc)._name__}: {exc}"}
            conn.sendall((json.dumps(reply) + "\n").encode("utf-8"))

    sink.close()
    server.close()
    sock_path.unlink(missing_ok=True)
    pidfile.unlink(missing_ok=True)

if name == "__main__":
    main()

The small client gives every command a synchronous acknowledgment barrier:

#!/usr/bin/env python3
# ctl.py
import argparse, json, socket, sys

ap = argparse.ArgumentParser()
ap.add_argument("socket")
ap.add_argument("command", nargs="+")
a = ap.parse_args()

s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.connect(a.socket)
s.sendall((" ".join(a.command) + "\n").encode())
reply = b""
while not reply.endswith(b"\n"):
    chunk = s.recv(4096)
    if not chunk:
        break
    reply += chunk
s.close()

text = reply.decode().strip()
print(text)
try:
    obj = json.loads(text)
except json.JSONDecodeError:
    sys.exit(2)
sys.exit(0 if obj.get("status") == "ACK" else 1)

Run writer.py directly in a dedicated terminal. It remains a foreground process; it neither daemonizes nor discovers other PIDs.

Capture both path identities and live descriptors

A pathname stat and a process-descriptor inspection answer different questions. Capture both.

Linux exposes a process's open descriptors as links under /proc/<pid>/fd, while /proc/<pid>/fdinfo/<fd> provides fields including descriptor position, flags, mount ID and inode. Access can depend on procfs permissions and mount options such as hidepid, so the fixture should run the writer and probes as the same test user and record any access failure rather than silently escalating privileges.

Use this probe:

#!/usr/bin/env bash
# probe.sh
set -euo pipefail
LAB=${1:?usage: probe.sh LAB LABEL}
LABEL=${2:?usage: probe.sh LAB LABEL}
PYTHON_BIN=${PYTHON_BIN:-python3.14}

read -r PID EXPECTED_START < <(
"$PYTHON_BIN" - "$LAB/writer.pid" <<'PY'
import json, sys
x=json.load(open(sys.argv[1]))
print(x["pid"], x["starttime_ticks"])
PY
)

CURRENT_START=$("$PYTHON_BIN" - "$PID" <<'PY'
from pathlib import Path
import sys
s=Path(f"/proc/{sys.argv[1]}/stat").read_text()
print(s.rsplit(")",1)[1].split()[19])
PY
)

[[ "$CURRENT_START" = "$EXPECTED_START" ]] || {
    echo "PID/start-time mismatch; refusing to inspect recycled PID" >&2
    exit 3
}

OUT="$LAB/probe-$LABEL.txt"
{
    echo "label=$LABEL"
    echo "captured_utc=$(date -u +%FT%TZ)"
    cat "$LAB/writer.pid"
    echo
    echo "[paths]"
    for p in "$LAB/app.log" "$LAB/app.log.1"; do
        if [[ -e "$p" ]]; then
            printf '%s ' "$p"
            stat -Lc 'mode=%a owner=%U group=%G dev=%d ino=%i size=%s nlink=%h' "$p"
        else
            echo "$p MISSING"
        fi
    done

    echo
    echo "[proc-fd-directory]"
    stat -Lc 'mode=%a owner=%U group=%G' "/proc/$PID/fd"
    ls -l "/proc/$PID/fd"

    echo
    echo "[log-related-fds]"
    for f in /proc/"$PID"/fd/*; do
        target=$(readlink "$f" 2>/dev/null || true)
        case "$target" in
          "$LAB"/app.log*)
            echo "fd=${f##*/} target=$target"
            stat -Lc 'dev=%d ino=%i size=%s nlink=%h' "$f"
            cat "/proc/$PID/fdinfo/${f##*/}"
            ;;
        esac
    done
} > "$OUT"

cat "$OUT"

The PID alone is insufficient as a long-lived identity because PIDs can eventually be reused. /proc/<pid>/stat field 22 is the process start time after boot, expressed in clock ticks on modern Linux; combining PID, that start value, and the boot ID makes the fixture much less likely to inspect a replacement process accidentally.

If a proc fd symlink visibly ends in (deleted), interpret that as evidence that the pathname/link relationship has gone away while the process still has an open descriptor. Do not interpret the word “deleted” as “the process has closed the file.” The actual acceptance evidence remains the descriptor's device/inode and fdinfo, not the label alone.

Use records and file identity as complementary evidence

The identity timeline establishes where the writer points. Sequence reconciliation establishes which declared records are present. Neither substitutes for the other.

An inode match after rotation can prove that the descriptor still references the old file object, but it cannot prove that sequences 1–6 all exist exactly once. Conversely, finding six lines does not prove that the post-rotation records went to the approved active pathname.

This is a narrower use of evidence than what operational observability can establish: dashboards and aggregate telemetry can help detect anomalies, but the present acceptance test is deliberately grounded in the local process handle and the raw fixture records.

Preserve, at minimum, environment.txt, the logrotate configuration, its hash, the private state file, the logrotate outputs and return codes, every probe-*.txt, the acknowledgment JSON lines and all retained app.log* files until the decision is complete.

Reproduce rename/create with a writer that never reopens

Start a fresh fixture, then run the custom writer in terminal A:

RUN_ID=rename-sticky

python3.14 "$LAB/writer.py" \
  --log "$LAB/app.log" \
  --socket "$LAB/control.sock" \
  --pidfile "$LAB/writer.pid" \
  --run "$RUN_ID" \
  --mode custom

In terminal B, create the pre-rotation range and record its acknowledgments:

: > "$LAB/acks.jsonl"

for n in 1 2 3; do
    python3.14 "$LAB/ctl.py" "$LAB/control.sock" emit "$n" |
      tee -a "$LAB/acks.jsonl"
done

"$LAB/probe.sh" "$LAB" before-rotation

Before rotation, the expected identity invariant is:

stat(app.log).dev,ino == stat(/proc/PID/fd/LOGFD).dev,ino

Then exercise debug mode.

Distinguish a dry run from an executed rotation

The logrotate 3.22.0 manual says --debug makes no changes to log files and does not update the state file. An exit-zero debug run therefore checks how logrotate evaluates the configuration; it is not evidence that a rotation occurred.

Capture it explicitly:

set +e
logrotate --debug --force \
  --state "$LAB/logrotate.state" \
  "$LAB/logrotate.conf" \
  >"$LAB/logrotate-debug.txt" 2>&1
DEBUG_RC=$?
set -e
printf '%s\n' "$DEBUG_RC" > "$LAB/logrotate-debug.rc"

"$LAB/probe.sh" "$LAB" after-debug

Now perform one real forced rotation:

set +e
logrotate --verbose --force \
  --state "$LAB/logrotate.state" \
  "$LAB/logrotate.conf" \
  >"$LAB/logrotate-run.txt" 2>&1
ROTATE_RC=$?
set -e
printf '%s\n' "$ROTATE_RC" > "$LAB/logrotate-run.rc"

"$LAB/probe.sh" "$LAB" after-rotation

The maintainer manual documents --force, an alternate state file through --state, and create, which constructs a new file under the original log pathname immediately after rotation and before any postrotate script.

Do not send reopen in this negative control. Emit the second declared range:

for n in 4 5 6; do
    python3.14 "$LAB/ctl.py" "$LAB/control.sock" emit "$n" |
      tee -a "$LAB/acks.jsonl"
done

"$LAB/probe.sh" "$LAB" after-post-rotation-emits

The documented mechanism predicts the following, but the prediction must not be substituted for the captured files:

The key Linux rule is documented by rename(2): renaming a pathname does not disturb already-open descriptors. Therefore an empty new active file following exit zero is entirely compatible with a healthy live writer that is simply still attached to the rotated inode.

Implement an explicit application reopen contract

The corrected custom case uses an application operation with a precise contract: reopen first opens the intended pathname, verifies that the new descriptor and pathname identify the same regular file, then swaps the live stream and closes the prior descriptor. If opening or validation fails, the old stream remains available.

That contract belongs to the application. A generic SIGHUP must never be assumed to mean “reopen logs.” The logrotate manual contains configuration examples that signal particular daemons, but those examples do not establish a universal signal contract for unrelated software.

Reset the lab and rerun prepare.sh; do not reuse the negative-control writer, state file or record files. Start writer.py --mode custom with a new run ID, emit 1–3, and rotate exactly as before.

Pause, rotate, reopen, acknowledge, then resume

The critical repaired ordering is:

# Sequences 1-3 were acknowledged before this point.

logrotate --verbose --force \
  --state "$LAB/logrotate.state" \
  "$LAB/logrotate.conf" \
  >"$LAB/logrotate-run.txt" 2>&1

"$LAB/probe.sh" "$LAB" rotated-before-reopen

python3.14 "$LAB/ctl.py" "$LAB/control.sock" reopen |
  tee -a "$LAB/acks.jsonl"

"$LAB/probe.sh" "$LAB" after-reopen

for n in 4 5 6; do
    python3.14 "$LAB/ctl.py" "$LAB/control.sock" emit "$n" |
      tee -a "$LAB/acks.jsonl"
done

"$LAB/probe.sh" "$LAB" after-repaired-emits

There is no arbitrary sleep. Completion of the rotation command is one barrier; the synchronous ACK reopen is the next; only then does the harness permit sequence 4.

The expected repaired routing is unambiguous: 1–3 belong in app.log.1; 4–6 belong in the replacement app.log. The after-reopen probe must show the writer's descriptor device/inode matching stat(app.log). Also inspect:

stat -Lc 'mode=%a owner=%U group=%G dev=%d ino=%i' "$LAB/app.log"

The expected mode is 0640, with the configured fixture user and group, because that is the create contract supplied to logrotate. The manual documents that create can specify mode, owner and group for the replacement.

This barrier-controlled handoff establishes only the tested application contract. It does not prove that a continuously writing multi-threaded production application, a different signal implementation, or a different filesystem can make the same transition without an application-specific concurrency design.

Compare inode-aware application logging separately

Python's WatchedFileHandler documentation for Python 3.14 describes a different application-side solution. On Unix/Linux, it checks whether the file has changed since the previous emission; a change includes a different device or inode. If changed, the old stream is closed and the filename is reopened before the record is emitted. The documentation specifically identifies tools such as logrotate as a motivation.

Reset the fixture again and start:

RUN_ID=watched-handler

python3.14 "$LAB/writer.py" \
  --log "$LAB/app.log" \
  --socket "$LAB/control.sock" \
  --pidfile "$LAB/writer.pid" \
  --run "$RUN_ID" \
  --mode watched

Emit 1–3, rotate, and probe before sending record 4:

"$LAB/probe.sh" "$LAB" watched-immediately-after-rotation

python3.14 "$LAB/ctl.py" "$LAB/control.sock" emit 4 |
  tee -a "$LAB/acks.jsonl"

"$LAB/probe.sh" "$LAB" watched-after-first-post-rotation-emit

for n in 5 6; do
    python3.14 "$LAB/ctl.py" "$LAB/control.sock" emit "$n" |
      tee -a "$LAB/acks.jsonl"
done

The timing is important. Immediately after an external rename/create rotation but before another application emission, the handler can still have its old stream. The documented check occurs in connection with emission; therefore the decisive descriptor probe is also taken after sequence 4.

That differs from the explicit handshake. The custom protocol lets an external operator demand and receive an ACK reopen before record production resumes. WatchedFileHandler instead performs its inode/device-aware check as part of its own emit path.

Do not turn inode detection into a universal guarantee

Python documents WatchedFileHandler as intended for Unix/Linux and says it is not appropriate for Windows, where its documented assumptions differ and st_ino support does not provide the same mechanism.

The fixture consequently accepts only this scoped proposition: with one Python 3.14 writer, the recorded Linux filesystem/mount, the exact handler configuration above, and rename/create rotation, inspect whether the first post-rotation emission causes the live handler stream to move to the new device/inode.

Do not extrapolate that observation to multiple worker processes, network or unusual shared filesystems, copy/truncate behavior, external truncation, application crash recovery, or collector delivery. Those are different contracts.

Explain the copytruncate window without fabricating observations

copytruncate solves a different problem by leaving the application's original pathname/inode in place. The logrotate 3.22.0 manual says that this mode copies the original log and then truncates the original file to zero instead of moving it. It also explicitly warns that a small interval exists between copy and truncate during which logging data can be lost.

That warning is the acceptance boundary.

Suppose the source contains sequences 1–100 when copying begins. If the application appends sequence 101 after the copier has passed the relevant end-of-file boundary but before logrotate truncates the source, 101 can fall into the documented vulnerable interval: it may be absent from the completed copy and then removed by truncation. The exact timing is implementation and workload dependent; the important point is that the maintainer documentation itself says loss is possible.

Accordingly, do not write:

“We ran copytruncate repeatedly and observed no missing sequence IDs, therefore copytruncate is lossless.”

At most, such a run supports:

“No missing fixture IDs were observed in the specific executed range.”

Those are materially different statements.

The same caution applies in the opposite direction. This playbook does not claim that a normal logrotate invocation can be manipulated to force the race every time. An artificial barrier-controlled “copy, append, truncate” demonstration could illustrate the mechanism, but it would model the documented race rather than reproduce logrotate's internal scheduling faithfully.

copytruncate may still be an operational choice when an application cannot reopen its log. The acceptance record should simply carry the documented loss-window limitation rather than hiding it behind a successful rotation exit code.

Reconcile every expected record across the retained files

Byte counts are not enough. Six expected records could produce six lines while containing a duplicate sequence and a missing one. Conversely, an empty active file can coexist with a complete record set in app.log.1.

Use an exact parser:

#!/usr/bin/env python3
# reconcile.py
import argparse
from collections import defaultdict
import json
from pathlib import Path
import re
import sys

REV = "refonte-logrotate-fd-v1"
LINE = re.compile(
    r"^run=(?P<run>[A-Za-z0-9._-]+) "
    r"seq=(?P<seq>[0-9]{6}) "
    r"fixture=refonte-logrotate-fd-v1$"
)

def expected_set(spec):
    out = set()
    for item in spec.split(","):
        if "-" in item:
            a, b = map(int, item.split("-", 1))
            out.update(range(a, b + 1))
        else:
            out.add(int(item))
    return out

ap = argparse.ArgumentParser()
ap.add_argument("--run", required=True)
ap.add_argument("--expected", required=True)
ap.add_argument("--active", required=True)
ap.add_argument("--post-start", type=int, required=True)
ap.add_argument("files", nargs="+")
a = ap.parse_args()

expected = expected_set(a.expected)
locations = defaultdict(list)
invalid = []
wrong_run = []
active = str(Path(a.active).resolve())

for name in a.files:
    path = Path(name)
    if not path.is_file():
        continue
    try:
        lines = path.read_text(encoding="utf-8").splitlines()
    except Exception as exc:
        invalid.append({"file": str(path), "error": str(exc)})
        continue

    for number, text in enumerate(lines, 1):
        m = LINE.fullmatch(text)
        if not m:
            invalid.append({"file": str(path), "line": number, "text": text})
            continue
        if m.group("run") != a.run:
            wrong_run.append({"file": str(path), "line": number,
                              "run": m.group("run")})
            continue
        seq = int(m.group("seq"))
        locations[seq].append(str(path.resolve()))

present = set(locations)
missing = sorted(expected - present)
unexpected = sorted(present - expected)
duplicates = {
    str(seq): locs for seq, locs in locations.items() if len(locs) != 1
}
routing_failures = {
    str(seq): locs
    for seq, locs in locations.items()
    if seq in expected and seq >= a.post_start and
       (len(locs) != 1 or locs[0] != active)
}

result = {
    "run": a.run,
    "expected": sorted(expected),
    "missing": missing,
    "duplicates": duplicates,
    "unexpected": unexpected,
    "invalid_lines": invalid,
    "wrong_run_records": wrong_run,
    "routing_failures": routing_failures,
    "locations": {str(k): v for k, v in sorted(locations.items())},
}

print(json.dumps(result, indent=2, sort_keys=True))

ok = not any([
    missing, duplicates, unexpected, invalid,
    wrong_run, routing_failures
])
sys.exit(0 if ok else 1)

Run it only against the retained files belonging to the fixture:

python3.14 "$LAB/reconcile.py" \
  --run "$RUN_ID" \
  --expected 1-6 \
  --active "$LAB/app.log" \
  --post-start 4 \
  "$LAB"/app.log "$LAB"/app.log.[0-9]* \
  > "$LAB/reconciliation.json"

printf 'reconciliation_rc=%s\n' "$?"

In the negative rename/create control, the expected reconciliation has missing=[] and no duplicates but routing failures for 4, 5 and 6 because their location is app.log.1. That distinction is the answer to the central diagnostic question: the records can survive the rename operation locally while still violating the active-file routing contract. The survival statement is valid only after inspecting the actual retained records from the executed run.

For the explicit-reopen and WatchedFileHandler cases, acceptance instead expects sequences 1–3 in the rotated file and 4–6 in active app.log, with no missing, duplicate, invalid, foreign-run or unexpected sequence entries.

Handle reopen failure without discarding the old descriptor

A useful reopen implementation must define what happens when the intended new pathname cannot be opened. The custom fixture deliberately chooses a recoverable design: candidate first, swap second.

Test that property inside the temporary directory. After a rename/create rotation, make the new active path temporarily unusable without altering the old writer descriptor:

mv "$LAB/app.log" "$LAB/app.log.candidate"
mkdir "$LAB/app.log"

set +e
python3.14 "$LAB/ctl.py" "$LAB/control.sock" reopen |
  tee -a "$LAB/acks.jsonl"
REOPEN_RC=$?
set -e

printf 'failed_reopen_rc=%s\n' "$REOPEN_RC" \
  >> "$LAB/reopen-failure.txt"

"$LAB/probe.sh" "$LAB" after-failed-reopen

Opening a directory as an appendable regular logfile causes this fixture's reopen validation to fail. The exact error must be retained rather than replaced with an expected transcript.

Because CustomSink.reopen() does not close old until after the candidate is successfully opened and validated, the fixture's prior sink remains available. That is a property of this deliberately implemented writer, not something to assume about another language runtime, logging framework or daemon.

Keep a recovery route when the new file is unusable

Restore only the fixture's intended target:

rmdir "$LAB/app.log"
mv "$LAB/app.log.candidate" "$LAB/app.log"

python3.14 "$LAB/ctl.py" "$LAB/control.sock" reopen |
  tee -a "$LAB/acks.jsonl"

"$LAB/probe.sh" "$LAB" after-reopen-recovery

Do not delete the old rotated file to “make the application notice.” Linux's descriptor semantics mean pathname manipulation is not a reliable way to make an already-open descriptor close; the application must close or replace that descriptor itself.

For an operational incident, record the last successfully acknowledged sequence before the failed reopen, the exact error, old descriptor device/inode, active-path identity, and subsequent successful ACK reopen. Until that chain is complete, stop promotion of the rotation procedure.

Do not solve an analogous production failure by broadly weakening directory permissions. Diagnose why the application's documented service identity cannot open the intended target and repair that contract specifically.

Make the automation job depend on evidence, not just exit zero

The logrotate manual states that errors cause non-zero status. That makes the return code useful evidence about the rotation utility's own execution. It does not document an inspection of every application's post-rotation descriptor, and therefore exit zero cannot by itself establish that a writer switched to the replacement inode.

The negative control is intentionally designed to demonstrate that separation: logrotate can successfully rename and create the requested files while the writer continues using the old descriptor, a behavior consistent with Linux rename(2).

An automated acceptance job should therefore retain a compact evidence ledger:

That is also why configuration automation as a separate layer should remain separate from runtime acceptance: successfully applying a configuration is not equivalent to proving an application's open descriptor changed.

Automation should expose both statuses, for example rotation_task=passed and continuity_acceptance=failed, rather than collapsing them into one green result.

Recover records and restore the approved routing path

When the active file is empty after rotation, first stop uncontrolled cleanup or additional rotations. Do not immediately truncate, compress, delete, or overwrite the suspected old file.

The recovery order is:

1.       Identify the live writer by PID plus recorded start identity.

2.       Capture its descriptor device/inode and the current active/rotated path identities.

3.       Retain the file behind the old descriptor.

4.       Establish a valid replacement app.log.

5.       Invoke only the application's documented reopen mechanism.

6.       Require the corresponding acknowledgment.

7.       Verify the new descriptor matches the active path.

8.       Reconcile the complete declared sequence before deleting anything.

This approach is consistent with the broader operational principle of preserving evidence before remediation discussed in Refonte Learning's article on incident evidence and controlled recovery, although that article addresses cybersecurity incident response rather than this specific logrotate mechanism.

If sequence 4 is found only in the retained rotated file, it is misrouted but recoverable. The appropriate recovery artifact can be derived from that retained evidence according to the application's downstream contract. Do not blindly concatenate the entire rotated file into app.log; that can duplicate records that are already present elsewhere.

If an expected sequence is absent from every retained file, acknowledgment record and other in-scope local artifact, classify it as unresolved loss within this test. Do not manufacture the missing record from a neighboring timestamp or sequence number.

The boundary is especially important for copytruncate: once data has actually fallen into its documented copy/truncate loss window and no independent retained source contains the bytes, this playbook has no basis for reconstructing them.

Choose accept, repair-reopen, recover-records, hold or revert

The final decision should describe the problem that remains, not merely whether logrotate exited successfully.

repair-reopen and recover-records are deliberately separate. Switching the writer to the new file does not move records already written to app.log.1. Conversely, recovering misplaced records does not fix the writer if its descriptor still targets the old inode.

revert is also not a command to rename files backward indiscriminately. Once a running process and filesystem have advanced, blind reversal can create new ambiguity. Revert means restore the previously approved configuration or operational method through a controlled change, keep the evidence, and re-establish a known writer path.

Define the evidence required for zero-loss claims

A defensible statement is:

“For run reopen-001, expected sequences 1–6 were each present exactly once in the retained files; sequences 4–6 were routed to the replacement active inode after an acknowledged reopen.”

A statement such as “logrotate is lossless” is not supported.

The minimum evidence for a zero-missing/zero-duplicate statement is the declared run and sequence range, raw retained record files, the writer acknowledgments, environment/configuration manifest, descriptor/path identity timeline and reconciliation output. The claim ends at that boundary.

Even a perfect six-record run does not establish crash durability, remote shipping, collector delivery, arbitrary concurrency behavior, other filesystem semantics, or the absence of a copytruncate race in future runs. Python's WatchedFileHandler documentation establishes its device/inode-aware reopening behavior; logrotate documents its own rotation modes; neither source makes a universal production zero-loss guarantee.

That restraint is not semantic caution for its own sake. It prevents a local, controlled acceptance result from being promoted into a much broader reliability promise than the evidence supports.

Assign writer, rotation and retention ownership

Reliable rotation crosses three operational responsibilities.

The application owner owns the writer contract: whether log files remain open, what command or API safely reopens them, what acknowledgment means, how reopen failure behaves, and whether multiple workers require coordination.

The system administrator owns the rotation contract: exact logrotate configuration, user/group context, creation mode, state-file behavior, rotation ordering and the local evidence necessary to prove the filesystem transition.

The reviewer or service owner owns approval and cleanup: deciding whether the observed record/routing evidence meets the service requirement and when retained files can safely move into the normal retention lifecycle.

That division fits within broader system-administration responsibilities, which include Linux operations, troubleshooting, recovery and controlled system changes rather than treating a configuration-file edit as the entire operational task.

A concise handover can be recorded as:

Fixture/config revision:
Environment manifest:
Application reopen interface:
Expected pre-rotation sequences:
Expected post-rotation sequences:
Old path dev/inode:
New active dev/inode:
Writer fd dev/inode after handoff:
Reopen acknowledgment:
Missing IDs:
Duplicate IDs:
Misrouted retained IDs:
Decision:
Evidence retention owner:
Revalidation trigger:

Escalate or hold whenever the writer contract is undocumented, the application acknowledgment cannot be obtained, /proc evidence cannot safely identify the intended process, sequence reconciliation is incomplete, or filesystem semantics differ materially from the approved test.

Revalidate after changes to the application runtime, logging handler, process model, filesystem/mount, privilege model, logrotate package/configuration, or reopen implementation. This is a log lifecycle contract; it is not a startup-readiness probe or a general statement of continuous service health.

Develop the administration foundations behind reliable rotation

This exercise depends on foundational Linux administration skills: command-line inspection, users and permissions, process state, filesystem identity, configuration control, troubleshooting and recoverable change execution. Refonte Learning's verified System Administration programme currently describes a six-month programme at 10–12 hours per week, with basic computer/network knowledge as the prerequisite and prior IT coursework helpful rather than mandatory; it also lists practical projects and completion certificates.

The programme page does not establish that it specifically teaches logrotate internals, /proc inode diagnostics, Python logging handlers, or this laboratory, so those specialist topics should not be implied.

For this playbook, the acceptance criterion remains concrete: identify the inode to which the live writer is actually appending, require an acknowledged switch when the application contract demands one, and reconcile every declared sequence across the retained files before calling rotation successful.