Deleting a flock File Can Let
Two Jobs Hold Different Locks

A fresh contender can successfully acquire flock() on job.lock while an earlier job still holds an exclusive lock on the object it opened. That observation sounds like a kernel failure only if we skip the filesystem-identity question. For the local Linux files covered here, flock() applies an advisory lock to an open file, while a pathname is how a later open() resolves a name to an object. Replacing or unlinking that name does not retarget descriptors that already refer to the old object. The result can therefore be two valid exclusive locks on two different file objects, even though both participants used the same pathname string at different times. Linux flock(2) and open(2) document the two contracts that explain this behavior.

This playbook covers cooperative, single-host jobs using private local filesystems and three pathname-lifecycle protocols: a stable pathname, unlink-and-recreate, and destination replacement. It excludes NFS/SMB, distributed leases, fcntl byte-range locks, pidfiles, daemonization, and inherited-descriptor lifetime. A run of the fixture below on 20 October 2026 produced PASS_15_CASES using CPython 3.12.14, Linux 6.18.55, and overlayfs. These are laboratory results; production acceptance also requires the application and maintenance controls described below.

The operational goal is not merely to reproduce the behavior. It is to decide whether a job-lock protocol is admissible, repair unsafe lifecycle rules, rerun after relevant environmental changes, and hold when evidence is incomplete or prior overlap cannot be reconciled. A diagnostic PASS can prove that an unsafe design fails exactly as predicted; it does not authorize that design.

Define the exclusion contract for the lock object

The contract for cooperative mutual exclusion is stronger than “every job opens the same string.” For the entire interval in which holders or contenders may exist, every cooperating participant must address the same intended local file object and then request compatible flock() semantics on that object. For these local files, the lock is advisory: a process with sufficient permissions can ignore the convention and perform I/O anyway. The lock therefore works because all participating jobs agree to use it. The flock(2) manual states that locks are associated with open file descriptions, which is why pathname identity must be kept separate from lock ownership.

A pathname is a lookup instruction in a namespace. open() resolves that name and returns a file descriptor, which is a process-visible handle referring to an open file description in the kernel. Crucially, open(2) states that an already-created reference is unaffected if the pathname is later removed or changed to refer to another file. The path can therefore stay textually identical while the object reached by a later open changes.

That gives us the failure model: job A opens object O1 through job.lock and locks O1. Maintenance then removes or replaces the directory entry. Job B later opens the same spelling, but the name now resolves to object O2. B can lock O2 without violating A's lock on O1. The kernel has not granted two exclusive locks on one file object; the application has accidentally created two lock domains.

For broader vocabulary around processes, permissions, filesystems and scheduled operations, the adjacent guide on Linux administration foundations is useful context, but it does not establish this pathname-replacement behavior. Its role here is foundational, not normative.

A review should therefore state the application invariant in object terms: one durable lock object, continuously reachable by the agreed pathname, for as long as any holder or waiter may still participate. The presence of an empty .lock file is not evidence that a lock is held, and the absence or age of a pathname is not evidence that destroying it is safe.

Separate namespace replacement from descriptor inheritance

This page answers a different question from the earlier article on inherited descriptors and flock lifetime. That experiment keeps each tested file object’s identity stable and examines when references to one shared open file description are closed after fork(). Here, the holder is launched as a separate process that opens the lock itself, and the parent later performs fresh opens after changing what the pathname names. The distinction matters because a same-object lifetime problem and a changed-object namespace problem require different repairs.

Three terms are enough for the rest of the playbook. A pathname such as /run/myjob/job.lock is a name resolved through the filesystem namespace. A file descriptor is the small integer handle a process receives from open(). An open file description is the kernel object created by open() and referenced by that descriptor; on Linux, flock() locks are associated with these open file descriptions. A fresh open() produces a fresh open file description, although it may still refer to the same underlying file object. open(2) and flock(2) document those relationships.

For file-object identity in this experiment, we record (st_dev, st_ino) from fstat(). The stat result fields identify the device and inode, so the pair is more useful than an inode number alone. We also record st_nlink, the hard-link count, because unlinking the last directory entry can reduce the original object’s link count to zero while an open descriptor still keeps that object alive. unlink(2) documents that lifetime. These comparisons concern the freshly created files and contemporaneous observations in this fixture; filesystem-specific identity limits are discussed below.

Keep the old lock alive without passing it to a child

The ordering in the fixture is intentional. The parent first creates a private pathname, then starts a new Python process with subprocess.Popen. Only inside that separately launched process does child(path) call os.open() and fcntl.flock(). The parent never opens a holder descriptor before process creation, so there is no lock descriptor available to inherit. close_fds=True reinforces that the child should not receive unrelated parent descriptors.

This ordering is part of the evidence. If a rewritten test opened and locked the file in the parent before a fork(), it would drift back into the earlier same-open-file-description problem. A valid rerun of this protocol must preserve independent holder creation first, then later fresh probes and namespace operations.

Create a private local-filesystem laboratory

The fixture creates one TemporaryDirectory with three subdirectories named stable, unlink, and replace. Every lock file, replacement file, child process, descriptor and cleanup action belongs to that temporary root. Nothing in the procedure requires root privileges, a production scheduler, or access to a real job lock. That containment is a hard safety boundary: never substitute an operational lock pathname to “see what happens.”

The script requires GNU stat on PATH. Before any arm runs, it executes stat -f -c %T against its private root and accepts only an explicit filesystem allowlist: overlayfs, tmpfs, ext2/ext3, ext4, xfs, or btrfs. An unrecognized filesystem raises an error beginning HOLD: unqualified filesystem. This is a preflight filter, not proof that every configuration of a listed filesystem is qualified or that every omitted local filesystem is unsafe. The mount and backing-storage configuration must also fit the intended local scope. Network filesystems are excluded; the Linux flock(2) documentation describes materially different NFS and SMB behavior, which requires a separate qualification plan.

The script also bounds protocol waits to three seconds. selectors.DefaultSelector() waits for child output; record size is capped; unexpected EOF fails; and subprocess.wait(timeout=TIMEOUT) bounds shutdown. These are not performance expectations. They are test-orchestration limits so that a broken handshake becomes an explicit inconclusive run rather than a sleep-based guess.

The run reported here used CPython 3.12.14 on Linux kernel 6.18.55 with overlayfs. The Python 3.12 fcntl documentation describes fcntl.flock(fd, operation) as an operation on a file descriptor and states that failures raise OSError. The Linux flock(2) contract supplies the platform-specific semantics, including the use of LOCK_NB with a lock mode for nonblocking acquisition. These sources support the mechanism; the recorded fixture output establishes what happened in this particular run.

Treat the environment record as part of reproducibility. On a different Python build, kernel, filesystem or container/storage stack, rerun rather than borrowing the earlier PASS. Device and inode numbers are run-specific observations, never expected constants.

Run the complete three-arm lock protocol

Save the following fixture as flock_path.py using UTF-8, LF line endings, and a final newline. The SHA-256 of this exact source is 90c624189dff55f958a1135140a03950a7d147a6bb370fcd7b28bce1aa99bb9a. The command writes structured results to flock-result.json in the current working directory and overwrites that file if it already exists. Choose an unused output filename when retaining earlier results. The fixture itself creates and removes only its own temporary tree.

python3 flock_path.py > flock-result.json

"""Private Linux lock-path replacement experiment; never targets a real job lock."""
import errno
import fcntl
import json
import os
from pathlib import Path
import platform
import selectors
import subprocess
import sys
import tempfile
import time
TIMEOUT = 3.0
def require(ok, message):
    if not ok:
        raise RuntimeError(message)
def identity(fd):
    s = os.fstat(fd)
    return {"device": s.st_dev, "inode": s.st_ino, "links": s.st_nlink}
def same_file(a, b):
    return (a["device"], a["inode"]) == (b["device"], b["inode"])
def emit(value):
    print(json.dumps(value), flush=True)
def child(path):
    fd = os.open(path, os.O_RDWR)
    try:
        fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
        emit({"event": "READY", "held": True, **identity(fd)})
        for line in sys.stdin:
            command = line.strip()
            if command == "PING":
                emit({"event": "HELD", "held": True, **identity(fd)})
            elif command == "RELEASE":
                return  # finally closes the only descriptor owned by this child.
            else:
                raise RuntimeError("unexpected owned protocol command")
    finally:
        os.close(fd)
class Messages:
    def init(self, process):
        self.process, self.buffer = process, b""
    def receive(self):
        deadline = time.monotonic() + TIMEOUT
        with selectors.DefaultSelector() as selector:
            selector.register(self.process.stdout, selectors.EVENT_READ)
            while b"\n" not in self.buffer:
                remaining = deadline - time.monotonic()
                require(
                    remaining > 0 and selector.select(remaining),
                    "handshake timeout"
                )
                part = os.read(self.process.stdout.fileno(), 4096)
                require(part, "unexpected child EOF")
                self.buffer += part
                require(len(self.buffer) < 8192, "oversized owned protocol record")
        line, self.buffer = self.buffer.split(b"\n", 1)
        return json.loads(line)
    def send(self, command):
        self.process.stdin.write((command + "\n").encode())
        self.process.stdin.flush()
def open_probe(path):
    fd = os.open(path, os.O_RDWR)
    details = identity(fd)
    try:
        fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
        return fd, {"acquired": True, **details}
    except OSError as error:
        os.close(fd)
        if error.errno not in (errno.EAGAIN, errno.EWOULDBLOCK):
            raise
        return None, {"acquired": False, **details}
def stop_owned(process):
    if process.poll() is None:
        process.terminate()
        try:
            process.wait(timeout=TIMEOUT)
        except subprocess.TimeoutExpired:
            process.kill()
            process.wait(timeout=TIMEOUT)
    for pipe in (process.stdin, process.stdout, process.stderr):
        pipe.close()
def protocol(root, mode):
    directory = root / mode
    directory.mkdir()
    path = directory / "job.lock"
    path.write_bytes(b"original owned lock object\n")
    p = subprocess.Popen(
        [sys.executable, "-u", str(Path(__file__).resolve()),
         "--hold", str(path)],
        stdin=subprocess.PIPE, stdout=subprocess.PIPE,
        stderr=subprocess.PIPE, close_fds=True
    )
    wire, retained, rows = Messages(p), [], []
    try:
        ready = wire.receive()
        require(ready["event"] == "READY", "missing READY")
        fd, baseline = open_probe(path)
        if fd is not None:
            retained.append(fd)
        require(
            not baseline["acquired"] and same_file(ready, baseline),
            "baseline lock missing"
        )
        rows.append({
            "id": mode + "_1_baseline", "holder": ready, "probe": baseline
        })
        if mode == "unlink":
            path.unlink()
            path.write_bytes(b"new owned lock object\n")
        elif mode == "replace":
            replacement = directory / "replacement.lock"
            replacement.write_bytes(b"replacement owned lock object\n")
            os.replace(replacement, path)
        fd, after = open_probe(path)
        if fd is not None:
            retained.append(fd)
        wire.send("PING")
        held = wire.receive()
        changed = mode != "stable"
        require(
            held["event"] == "HELD" and held["held"] and same_file(ready, held),
            "holder changed"
        )
        require(
            after["acquired"] == changed, "unexpected post-operation acquisition"
        )
        require(
            same_file(held, after) != changed, "unexpected current file identity"
        )
        require(
            held["links"] == (0 if changed else 1), "unexpected original link count"
        )
        rows.append({
            "id": mode + "_2_path_operation",
            "original_holder": held, "current_probe": after
        })
        other_fd, other = open_probe(path)
        if other_fd is not None:
            retained.append(other_fd)
        require(not other["acquired"], "current file lost its exclusion")
        rows.append({"id": mode + "_3_current_file_contends", "probe": other})
        for locked_fd in retained:
            os.close(locked_fd)
        retained.clear()
        wire.send("PING")
        still = wire.receive()
        probe_fd, released_b = open_probe(path)
        if probe_fd is not None:
            retained.append(probe_fd)
        require(
            still["held"] and released_b["acquired"] == changed,
            "premature original release"
        )
        rows.append({
            "id": mode + "_4_second_holder_closed",
            "original_holder": still, "probe": released_b
        })
        for locked_fd in retained:
            os.close(locked_fd)
        retained.clear()
        wire.send("RELEASE")
        require(
            p.wait(timeout=TIMEOUT) == 0,
            "child failed: " + p.stderr.read().decode()
        )
        final_fd, final = open_probe(path)
        if final_fd is not None:
            retained.append(final_fd)
        require(final["acquired"], "final owned file unavailable")
        rows.append({"id": mode + "_5_original_closed", "probe": final})
        return rows
    finally:
        for fd in retained:
            os.close(fd)
        stop_owned(p)
def main():
    require(sys.platform == "linux", "HOLD: this protocol is Linux-specific")
    with tempfile.TemporaryDirectory(prefix="refonte-flock-path-") as name:
        root = Path(name)
        fs = subprocess.check_output(
            ["stat", "-f", "-c", "%T", name], text=True, timeout=TIMEOUT
        ).strip()
        require(
            fs in {"overlayfs", "tmpfs", "ext2/ext3", "ext4", "xfs", "btrfs"},
            "HOLD: unqualified filesystem " + fs
        )
        rows = [
            row for mode in ("stable", "unlink", "replace")
            for row in protocol(root, mode)
        ]
    print(json.dumps({
        "status": "PASS_15_CASES", "python": platform.python_version(),
        "kernel": platform.release(), "filesystem": fs, "cases": rows
    }, indent=2))
if name == "__main__":
    if len(sys.argv) == 3 and sys.argv[1] == "--hold":
        child(sys.argv[2])
    else:
        main()

The protocol is ordered. READY is emitted after the independently launched child has opened the current file and acquired its exclusive lock. PING asks that child to report identity with fstat() and acknowledge its controlled code path, which contains no unlock or close before RELEASE. The HELD message is therefore evidence together with that history; it is not an independent kernel lock-status query. RELEASE returns from the holder function so its finally block closes the descriptor. Every contender uses a fresh os.open() followed by nonblocking LOCK_EX | LOCK_NB. The Python documentation confirms that fcntl.flock() failures surface as OSError, and the Linux manual specifies EWOULDBLOCK for nonblocking contention. The fixture accepts EAGAIN/EWOULDBLOCK in its Linux-only scope and re-raises other errors.

The fixture does not use sleeps to infer overlap. It waits for explicit messages that establish causal order: holder ready, namespace operation, probe, holder acknowledgement, second probe, close, further acknowledgement, release. That is stronger than saying “the child has been running for five seconds, so it probably still holds the lock.” Process age is not ownership evidence.

Make every failed handshake an inconclusive run

An expected lock-contention result is narrow: the fresh flock() fails with the recognized contention errno. Everything else is a protocol failure. A timeout, malformed JSON record, unexpected EOF, child nonzero exit, identity mismatch, wrong link count, surprising acquisition result, or unfamiliar filesystem must stop the interpretation. Do not downgrade such failures to “probably blocked.”

The finally path closes only descriptors retained by this run and invokes stop_owned() only for the exact Popen child created by the fixture. It never scans PIDs, kills arbitrary processes, or deletes a path guessed to be stale. This is an important operational habit to carry into real incident work: uncertain ownership should lead to HOLD and evidence gathering, not broad cleanup.

Prove the stable pathname blocks a second owner

The stable arm is the negative control for namespace change and the positive control for ordinary release. The holder opens stable/job.lock, acquires an exclusive lock, and sends READY. At stable_1_baseline, a fresh probe opens the same current pathname and must fail to acquire. The fixture also requires the holder's (device, inode) pair to match the fresh probe's pair, so the blocked acquisition is tied to the same file object rather than merely the same string.

No namespace operation occurs in stable mode. At stable_2_path_operation, another fresh probe remains blocked, the holder still reports the same identity it reported at READY, and the original object's link count remains one. This is the control that prevents us from interpreting “fresh opens always succeed” or “this filesystem ignores flock().” Linux documents that an exclusive lock conflicts with another exclusive lock on the same file, and that distinct opens are treated independently for flock() purposes; this arm exhibits that documented contract in the recorded environment. See flock(2).

At stable_3_current_file_contends, the fixture makes another fresh open while no replacement holder has been created. It must still be blocked. At stable_4_second_holder_closed, the retained-probe list is empty, so its close loop does nothing. The original child acknowledges that it remains on its holding code path, and a new probe is still blocked. This checkpoint mirrors the changed arms while confirming that the original holder continues to exclude fresh opens of the stable object.

Only after RELEASE, when the original child returns and closes its only holder descriptor, does stable_5_original_closed acquire successfully. That sequence is the expected same-object lifecycle. The result is not surprising; its value is as a control against which unlink and replacement can be compared.

The script creates the regular file before the child locks it. That existing file may be available for locking or contended by another descriptor; its presence alone does not establish that a lock is held. In this private fixture, if the child has not acquired the lock and no other holder exists, a fresh contender can acquire it. The existence, mtime, content, or age of job.lock is therefore not a reliable basis for deleting a supposedly stale lock file.

Unlink the name while retaining the original lock

The unlink arm starts identically: create unlink/job.lock, launch the child, let the child open and lock that file, and prove at unlink_1_baseline that a fresh open reaches the same device/inode and cannot acquire. Only then does the parent perform the controlled namespace change: path.unlink() removes the name and path.write_bytes(...) creates a new regular file under the same pathname.

The key documented behavior comes from unlink(2): unlink removes a name, and if that was the last link but a process still has the file open, the file remains in existence until the last descriptor referring to it closes. The holder descriptor therefore continues to address the original object even though no directory entry names it. open(2) independently explains why: the descriptor's reference is not retargeted when the pathname is removed or modified.

At unlink_2_path_operation, the parent opens the recreated pathname and reaches the newly created object. In the recorded run, the fresh probe acquired an exclusive flock(), and a subsequent PING returned the original child’s identity while its controlled code path continued holding the original lock. The fixture requires the (device, inode) pairs to differ and the original holder’s st_nlink to be zero. Together with the successful baseline and unchanged holder code path, those checks establish that the original object remains open and locked while the name reaches a different object.

That is the application failure. Two job instances could now each believe they hold the cooperative exclusion token. Yet the kernel's per-object lock behavior remains coherent. The first job holds a lock on O1; the new contender holds a lock on O2. There are not two exclusive flock() locks on one inode.

unlink_3_current_file_contends proves this orthogonally. The fixture deliberately retains the newly acquired descriptor on O2, then performs another fresh open of the current pathname. That third probe is blocked. In other words, O2 now has normal exclusion against other contenders that also reach O2. The broken property is not “flock stopped working”; it is that the application has split what it intended to be one lock domain into O1 and O2.

Zero link count does not mean the original holder is gone

The zero link count describes namespace reachability. fstat() on the child-held descriptor still returns the original device/inode and links: 0. The HELD acknowledgement, together with the controlled acquisition and release history, establishes that this child has not unlocked or closed the descriptor. The Linux unlink semantics permit the original object to persist because it remains open.

Do not try to “find the old lock again” by opening the removed pathname. That pathname now either does not exist or, in this fixture, resolves to the newly created O2. The evidence for O1 comes from the retained descriptor and controlled process history. Likewise, do not treat nlink == 0 as evidence that the lock is released. It means no ordinary directory entry links that inode; it says nothing by itself about whether a process still has it open.

The repair implication is direct: a cooperative lock object's pathname must not be part of ordinary stale-file cleanup, rotation or recreate-on-start logic while holders or contenders can exist. Deleting a lock file because it “looks old” is not recovery; it can manufacture a second lock domain.

Replace the destination name without a deletion gap

Operators sometimes replace files with a rename-based pattern to avoid a period when the destination pathname is missing. That is useful for many payload-update designs, but it does not preserve lock-object identity. In the replace arm, the fixture creates replacement.lock as a separate file in the same directory and calls os.replace(replacement, path). Python’s os.replace() documentation specifies replacement of an existing file when permissions permit and an atomic rename on success; the operation may fail across filesystems. The rename(2) manual explains the destination-visibility guarantee, while open(2) explains why existing descriptors are not retargeted.

The baseline proves ordinary contention first. Then replace_2_path_operation performs the replacement and immediately makes a fresh probe through job.lock. The current pathname reaches the prepared replacement inode. In the recorded run, that fresh probe acquired a lock. The child then answered HELD with the same original identity it had reported at READY, and the fixture required the original object’s link count to be zero. Its code had neither unlocked nor closed the original descriptor.

There is no separate unlink-and-create gap in this arm. A successful atomic rename means other lookups of the existing destination do not encounter a missing name as an intermediate state of that operation. This namespace guarantee does not migrate open descriptors from the previous destination to the replacement. The rename(2) and open(2) contracts distinguish those two properties.

Therefore “we update the lock file atomically” is not an admissibility argument. Atomicity of the rename and mutual exclusion across old and new inodes are different properties. A release/deploy mechanism that stages a new lock file and atomically renames it into place can split the lock domain just as effectively as unlink-and-recreate.

The third checkpoint shows that the new object behaves normally. replace_3_current_file_contends opens the current pathname while the fixture retains the acquired lock on the replacement object; the probe is blocked. The kernel enforces the replacement object’s exclusive lock while the original child continues holding a separate exclusive lock on the old, now-unlinked object.

Reconcile both live holders and a third fresh probe

Taken together, the changed arms establish three facts to record in a reproduction. First, the original child remains responsive and reports the same original device/inode after the pathname change, without an intervening unlock or close in its controlled code. Second, a fresh open of the current pathname reports a different device/inode and acquires an exclusive lock. Third, while that replacement descriptor is retained, an additional fresh open of the current pathname is blocked. Those facts distinguish object replacement from a failure to enforce contention on either object.

The evidence combines the original holder’s controlled history, a current pathname that resolves elsewhere, and successful contention on the current object. It supports the conclusion that the application’s intended mutual exclusion was broken by namespace replacement while exclusion on each individual object remained intact. The Linux manuals supply the mechanism: flock() applies to the open file; descriptor references survive pathname changes; unlink can leave an open unlinked object alive; and rename replacement changes the destination name without retargeting existing descriptors.

The recorded checkpoint results are:

The table summarizes the run reported above. Device numbers, inode numbers, and runtime metadata can differ on another run; they are not expected constants. The acceptance criteria concern relationships: same versus different identity, blocked versus acquired at named checkpoints, link count where asserted, complete ordered handshakes, and successful child exit. The JSON records those checkpoint observations and environment fields; it does not measure performance.

The neighboring article on hard-linked backup identity also uses (device, inode) as file identity evidence rather than trusting pathnames or content labels. Here that comparison helps explain why two job owners no longer contend on the same lock object. This lock fixture transfers or modifies no backup contents, and the backup workflow is contextual reading rather than evidence for the locking result.

Interpret device and inode evidence within its tested scope

Within this fresh-file fixture and the observed filesystem configuration, matching (device, inode) pairs support the same-object interpretation, and differing pairs support the changed-object interpretation. The original descriptor stays open during the decisive comparison. These values are not permanent identifiers across deletion and reuse, and they require filesystem-specific interpretation. The kernel’s OverlayFS documentation warns that st_dev and st_ino can change during a non-directory object’s lifetime under some configurations. A surprising identity result therefore requires investigation, not an automatic replacement diagnosis. The pair also does not establish whether two descriptors share one open file description; that requires descriptor provenance and process history.

The fixture establishes provenance because the holder performs its own os.open() in a separately launched process, while every probe performs a fresh os.open() later in the parent. Those calls create independent open file descriptions. In the inherited-descriptor experiment, the process history instead involves a shared open file description across fork(). Device/inode evidence addresses file identity here; it cannot by itself determine how descriptors were created or shared.

Close the replacement holder while the original remains alive

The fourth checkpoint is useful for operations. After unlink_3 or replace_3, the fixture closes every descriptor it retained on the current object O2. It then sends PING to the original child. That acknowledgement, together with the absence of an intervening unlock or close in the child’s code, establishes that the original descriptor still holds O1. Next the parent opens the current pathname again.

In the changed arms, unlink_4_second_holder_closed and replace_4_second_holder_closed acquire successfully. That is expected because the retained O2 holder has closed, so there is no incompatible lock on O2. The original O1 holder is irrelevant to this fresh open because the pathname no longer resolves to O1. This is precisely why repairing only the pathname after overlap is dangerous: observers of the current name may see an apparently available lock even while an older owner remains active on the orphaned object.

The stable arm provides the contrast. At stable_4_second_holder_closed, the original child still owns the same object reached by the pathname, so the fresh probe remains blocked. No namespace split means one lock domain still exists.

Finally, the fixture closes any newly retained current-object descriptor, sends RELEASE to the child, waits for a zero exit status, and performs the fifth probe. stable_5_original_closed, unlink_5_original_closed, and replace_5_original_closed all acquire. The meaning differs slightly: in stable mode the original/current object has finally been released; in changed modes the current object was already independent, and the checkpoint additionally proves that the owned original child exited cleanly. The five rows per arm are all necessary because a single midpoint observation cannot distinguish baseline failure, current-object contention, release order and final cleanup.

Repair the lifecycle around a stable lock object

The repair is architectural, not a clever cleanup command: give the lock a dedicated, persistent file object whose lifecycle is separate from replaceable job payloads, status files, deployment trees and generated configuration. Create it under a directory with an explicit operational owner and permissions, then leave that file object's directory entry stable across normal runs and deployments. Jobs may open and close descriptors; they must not unlink, rotate, truncate-by-replacement or atomically swap the lock target as part of routine operation.

Truncating the existing lock file through an already resolved descriptor is a different namespace question, but there is usually no operational reason to store mutable job state in the lock object at all. The cleaner design is an otherwise boring lock file used only as the rendezvous object. Put human-readable state, logs, last-run metadata and release artifacts elsewhere. This separation reduces pressure to replace the lock path just because another payload needs atomic update semantics.

Directory ownership and mode bits should restrict who can mutate the namespace. That supports the cooperative policy by preventing unrelated service accounts from casually replacing the lock entry. For the local Linux files covered here, flock() remains advisory: a process that ignores the protocol can still operate on protected payloads if their permissions allow it. Namespace permissions and participation in the locking protocol therefore address separate requirements.

A minimum application contract can be written as follows:

The key point is that maintenance tooling must treat the lock object as infrastructure identity, not disposable state. A release process may replace the executable, configuration or payload around it, but the lock pathname itself stays bound to the same file object for the participation lifetime.

Quiesce all participants before namespace maintenance

There are legitimate reasons to rebuild a lock directory, change ownership, migrate local storage, or redesign a scheduler. Those operations require a maintenance protocol, not opportunistic rm. Assign a named maintenance owner, stop new admission, and bring all known job launchers into a quiesced state. Then confirm that holders and waiters are gone using the application's authoritative orchestration or process/service controls. Only after that boundary may the maintenance owner replace or recreate the lock object according to the reviewed design.

The phrase “holders and waiters” is deliberate. A waiting contender may already have opened the lock object before it blocks. Replacing that object’s pathname can split existing and new participants across different objects, depending on how the application opens and waits. The supplied fixture uses nonblocking probes and does not execute a blocked-waiter replacement scenario. That operational extension needs its own controlled test before any claim about observed waiter scheduling.

The adjacent cleanup article on cleanup selection and protected paths reinforces a compatible discipline: destructive cleanup should be bounded to owned fixtures and verified against an explicit keep/remove contract. It does not provide a lock-recovery command. In this playbook, the production lock pathname belongs on the protected side of cleanup policy while the protocol is active.

A safe maintenance sequence is therefore:

1.      The job owner declares a maintenance window and the application-level state that must be quiescent.

2.      The automation owner disables all launch paths that could admit a new participant, including scheduled and release-triggered paths covered by the service's own inventory.

3.      The job owner confirms protected work is complete and no application-managed waiter remains pending.

4.      The filesystem/platform owner performs the namespace maintenance on the now-quiescent lock object or directory.

5.      The automation owner reinstates admission only after the repaired stable-object policy and permissions are verified.

6.      The job owner runs the application-specific post-maintenance validation and authorizes ACCEPT or RERUN.

This is intentionally not a list of shell commands that delete a “stale” lock. The safe action depends on ownership and application state. A generic rm job.lock is exactly the operation that can split the lock domain if an old descriptor is still alive.

A waiter can matter even when no job is currently inside its critical section

Suppose an implementation opens the lock object and then blocks in flock() while another holder owns it. That waiter may already have an open descriptor to O1. If maintenance replaces the pathname and new arrivals open O2, the system now has participants whose next actions depend on different objects. The precise wakeup and admission behavior should be tested for the actual implementation rather than guessed.

To test this extension, preserve the same discipline used here: a private local filesystem, explicit holder and waiter handshakes, recorded identities, bounded protocol waits, and no production path. Add the waiter scenario as a separate fixture with its own expected relationships. The present 15-case run does not cover it.

Reconcile work that may already have overlapped

Once evidence shows that two job instances could have held different lock objects, recreating the “right” lock file fixes only future admission. It cannot undo side effects that may already have occurred. Treat suspected overlap as an application-consistency event.

The job owner should define the protected effect set: files written, database operations invoked, releases promoted, queues consumed, external APIs called, emails sent, artifacts published, or any other action that the lock was meant to serialize. Then compare those effects against the job's own idempotency keys, manifests, transaction records, audit logs or reconciliation data. The correct recovery depends on the payload; there is no filesystem-level rollback that can universally reverse an arbitrary scheduled job.

Use REPAIR for the locking lifecycle when the namespace design can be corrected without disputing prior state. Use RERUN when the environment or implementation changed enough that the qualification evidence is stale, or when a repaired fixture needs to prove the corrected policy. Use HOLD when overlap may have occurred and application effects cannot yet be reconciled. Use REBUILD only where the application architecture has a defined rebuild procedure for derived outputs and the accountable data/job owner authorizes it. Do not use “the lock path exists again” as evidence that earlier effects are safe.

A practical incident ledger should record at least the suspected overlap interval, launch identifiers for each job instance, the lock-object identities observed if available, outputs or records that could have been touched, reconciliation status, and the person/team authorized to make the next decision. Avoid process-age heuristics: an old mtime or long-running PID does not prove which file object a participant has open or whether destructive cleanup is safe.

If the workload is designed to be idempotent, demonstrate that with its actual application evidence rather than asserting it from intent. If it is not safely repeatable, HOLD until a domain owner decides how to repair or compensate effects. Mutual exclusion protects application state only prospectively; once the exclusion contract has been violated, recovery belongs to the protected workload's consistency model.

Decide which locking protocol is admissible

The final decision separates the observed test result from deployment acceptance. The fixture produced PASS_15_CASES, but two of its three arms deliberately demonstrate unsafe application protocols. The PASS means all 15 checkpoint cases matched the expected relationships in the recorded CPython 3.12.14, Linux 6.18.55, and overlayfs environment. It does not approve unlinking or replacing a lock pathname while participants may still use it.

Operational ownership should be explicit. The job owner defines what mutual exclusion protects, the admission contract, and how overlap effects are reconciled. The automation/release owner ensures deployment, cleanup and scheduled workflows do not replace the lock target while participants may exist. The filesystem/platform owner qualifies the local storage environment and owns changes that can alter path or inode lifecycle. One team may hold several roles, but the responsibilities should still be named in the runbook.

The broader discussion of system automation provides background on repeatable configuration changes. For this locking protocol, the change owner must also distinguish stable lock objects from replaceable payloads. A repeatable workflow can repeatedly apply an unsafe replacement rule, so review the namespace effects of deployment and cleanup automation explicitly.

State the policy verdict alongside the test result

A commissioning summary should state the policy verdict beside the technical result. “PASS_15_CASES” by itself is misleading because a reader could assume every tested arm is acceptable. The stable arm is the candidate for ACCEPT within scope. The unlink/recreate and destination-replacement arms are successful failure reproductions: they prove why those lifecycle policies should be REJECTED or REPAIRED when active participants are possible.

A diagnostic that stops safely does not automatically approve an implementation. A filesystem rejection, timeout, or unexpected error means evidence is incomplete. Preserve the discrepancy, investigate, and rerun after the cause is understood. Do not change the fixture merely to obtain the preferred result.

Strengthen the administration skills behind safe job ownership

Reliable scheduled work depends on more than remembering a flock command. Administrators need enough Linux process, filesystem, permissions, networking, backup/recovery and automation knowledge to distinguish a pathname from an open object, to bound destructive maintenance, and to assign recovery decisions to the right owner. Those foundations are exactly what make a small lock-file incident reviewable instead of heuristic.

For structured practice in the wider skills behind this work, Refonte Learning’s System Administration Program lists Windows and Linux administration, file management, networking, security, troubleshooting, virtualization, backup and disaster recovery, command-line work, and practical projects. The page advertises 10–12 hours of weekly study and a Training Certificate and Certificate of Internship upon successful completion.

Keep the lock object stable, treat namespace replacement during participation as a protocol defect, and reconcile any work that may already have overlapped. Assign ACCEPT, REPAIR, RERUN, HOLD, REJECT, or REBUILD decisions to named owners who can assess the locking protocol and the protected workload’s state.