Imagine updating an OpenSSH server configuration with new Include files and overlapping Match blocks, then seeing a login still succeed. That is a misleading success if the client reused an old multiplexed connection or the daemon read different configuration bytes from the candidate you reviewed. This playbook defines a controlled acceptance decision for an owned disposable host. It inventories the installed binaries and service invocation, preserves a recovery route, follows the OpenSSH Include processing rules, calculates the Match criteria for each synthetic connection, and inspects the sshd -T -C effective configuration. It then requires fresh authentication and forwarding attempts correlated with server evidence before a proceed, hold, or restore decision. The commands, tables, and expected assertions below are a reproducible laboratory design, not observed results. A successful syntax check is structural evidence only; it does not prove PAM behavior, key-file permissions, network reachability, or a newly negotiated login.
Write the Remote-Access Contract First
Before editing sshd_config, document exactly who must connect, by which method, from which owned source, and with what forwarding rights. Use two synthetic accounts, alice and bob, plus a separate recovery identity. In the written examples, 198.51.100.10 and 203.0.113.20 are documentation-range labels; substitute addresses assigned to the isolated lab. The contract should distinguish public-key admission, password admission, local forwarding, remote forwarding, and any forced command. For example, alice from source A may use an approved key and local forwarding; alice from source B may use the key without forwarding; bob from source A may use a key without forwarding; bob from source B is denied; and sysadmin-recovery retains an independently approved key path. Name the change owner and recovery approver. Treat the policy as repeatable system-administration automation: version it, review it, and keep the proposal bounded to the change under review.
Stop condition: identify the designated console or recovery user before editing. Verify the out-of-band route, root or equivalent authority, the recovery key, the complete backup location, and the exact rollback command. Hold the change if any owner, credential, backup, restore step, or recovery dependency is unknown.
Inventory the Binary and the Running Service
Determine the exact SSH server and client builds on the host. Record the operating system, package name, package source, package version, executable paths, and locally supported version commands. On a Debian-family host, package metadata may come from dpkg-query or apt; on an RPM-family host, use the installed package manager. Capture ssh -V for the client and, when the installed build supports it, sshd -V for the server. Otherwise use package metadata for the server build instead of trying unsupported flags. The OpenSSH 10.5/10.5p1 release notes date that maintenance release to August 11, 2026, but this does not show that a particular distribution ships it. Record client and server builds separately because they may come from different packages or images.
Verify how the daemon actually starts. Inspect the running process and its parent, the service unit, any socket unit, container entry point, wrapper, and service-manager overrides. Commands such as ps, systemctl status, and systemctl cat can help on a systemd host, but use the tools appropriate to the installed service manager. Record every launch argument, including -f, -o, -p, and -h values, the active configuration path, host-key paths, PID behavior, and listening endpoints. The conventional /etc/ssh/sshd_config path is only a default; a command-line option or wrapper can select another file. Also record service drop-ins and whether a socket unit owns the listener. The sshd manual documents that command-line options can override configuration values, while the OpenSSH web manual index warns that its linked pages reflect the latest development release. Confirm every directive and Match criterion against the installed manuals. This inventory extends the operating-system and network foundations needed to review remote-access changes without assuming a universal distribution layout.
Preserve a Tested Recovery Route
Verify an out-of-band console or equivalent recovery channel before changing the service. Keep the current authenticated session open as a rollback aid, but label it pre-change evidence and do not use it to validate the new policy. Back up the complete active configuration tree, service overrides, and launch definition, not only sshd_config. A host-specific backup might begin with cp -a /etc/ssh /var/backups/ssh-before, followed by copies of relevant unit drop-ins or container definitions. Record the exact restore and reload or restart command, and verify that the recovery identity can use the separate route. Test the restore procedure on a disposable clone when practical. Never weaken authentication controls as a recovery technique. The plan must restore both configuration bytes and the service invocation that consumed them.
With recovery confirmed, editing may begin. Keep the old session outside the acceptance evidence and preserve it until fresh post-change access is proven. Hold if the console path, recovery identity, complete backup, restore command, service ownership, or authority to use the recovery route is missing.
Define the Stop Condition Before Editing
Hold the change if any safety dependency is unverified: no separate console, no working recovery key, no complete backup, no confirmed service invocation, or no tested restore command.
Follow Include Files in the Order Actually Read
OpenSSH configuration may be distributed across multiple files. Build a candidate manifest that lists each Include expression, every path produced by wildcard expansion, the lexical processing order, the canonical path, and a SHA-256 digest. The sshd_config manual states that wildcard matches are expanded and processed in lexical order, and that Include may appear inside a Match block for conditional inclusion. Record whether each include is global or conditional and which connection criteria make a conditional include active. A simple directory listing is not enough when paths are absolute, symlinked, or generated in a container namespace. Keep the candidate tree self-contained so the effective-policy test corresponds to the exact bytes proposed for deployment.
An absolute Include can silently mix candidate and live state. If /tmp/test-ssh/sshd_config includes /etc/ssh/common.conf, copying only the staged files does not isolate the test; the daemon still reads the live file. The live fragment may also change between validation and deployment. Replace the path with a candidate-owned location, or stage the entire referenced tree in an isolated filesystem namespace that resolves the same way during every test. The manifest must prove where every include resolves and which digest was read.
Reject a Mixed Candidate and Live Configuration
Reject a candidate whose Include graph escapes the approved staging root. For example, a staged /tmp/test-ssh/sshd_config that still names /etc/ssh/sshd_config.d/company.conf is mixed even if a copy of company.conf exists under /tmp. Resolve each literal path and wildcard using the same filesystem namespace that the test daemon will use, follow symlinks to canonical paths, and compare the result with the deployment manifest. sshd -T reports effective options; it does not provide a complete list of files opened, so grepping its output for paths cannot replace the Include manifest. After correcting an absolute path, regenerate the path list and digests, then rerun syntax and effective-policy checks. Validation is acceptable only when the tested bytes are the bytes that will be deployed.
Calculate Which Match Blocks Apply
For each connection case, resolve the explicit user, relevant primary and supplementary groups, source address, resolved source host name when used, local address, local port, and routing domain when supported. The sshd_config Match documentation lists the available criteria and identifies the subset of directives allowed inside a conditional block. Work from the complete Include-expanded order, because a matching conditional include can introduce additional directives. The compact fixture below illustrates overlapping scalar settings; it is not the complete admission policy for the four-case contract.
PasswordAuthentication no
PubkeyAuthentication yes
AllowTcpForwarding no
Match User alice Address 198.51.100.0/24
AllowTcpForwarding local
Match User alice
AllowTcpForwarding no
An alice connection from 198.51.100.10 satisfies both Match blocks. A satisfied Match block overrides the global value for an allowed keyword, and when the same keyword appears in multiple satisfied Match blocks, only the first matching instance is applied. The more specific first block therefore supplies AllowTcpForwarding local for source A; the later alice block does not replace it. An alice connection from 203.0.113.20 misses the address-specific block and receives AllowTcpForwarding no from the second block. Record the textual origin of each effective value so a reviewer can trace it to the manifest. This is not a universal last-line-wins rule, and list-valued directives may have keyword-specific append or override behavior.
Calculate the expected values for every case before running a command. Start with the global section, identify every satisfied Match block in processing order, and record PasswordAuthentication, PubkeyAuthentication, AllowTcpForwarding, ForceCommand, banners, and the separate account-admission decision. Where group matching matters, record the group-resolution input rather than assuming it. Apply the documented semantics for each keyword instead of treating every scalar and list option alike. This explicit matrix is security ownership beyond routine administration: it makes the expected user-and-address policy reviewable before a daemon is reloaded.
Use the following contract matrix as the expected-policy worksheet:
Connection case | Matching policy | Password | Public key / admission | TCP forwarding |
alice from source A | Specific alice+address block, then alice block | Denied | Allowed with synthetic approved key | Local only |
alice from source B | Alice block | Denied | Allowed with synthetic approved key | Denied |
bob from source A | Global values plus complete candidate admission policy | Denied | Allowed with synthetic approved key | Denied |
bob from source B | Complete candidate account/source denial | Denied | Login denied | Not available |
sysadmin-recovery | Independent approved recovery path | Denied | Allowed with recovery key | As separately approved |
The table separates configuration values from the live admission assertion. The compact Match fixture explains the forwarding overlap, while the complete candidate must also implement the bob-from-source-B denial and the independently approved recovery path with directives supported by the installed package. Add any locally relevant ForceCommand, PermitOpen, PermitListen, or group criteria to the worksheet rather than assuming they are absent.
Separate Syntax Checks From Effective Policy
Validate the candidate in two stages with the exact installed sshd binary and the same relevant invocation flags used by the service. First, perform the configuration and host-key sanity check:
sshd -t -f /tmp/test-ssh/sshd_config
echo "Exit status: $?"The sshd -t mode checks configuration validity and key sanity. Capture the command, standard error, and exit status. Any nonzero result stops the rollout. A zero result permits effective-policy inspection; it does not approve authentication or forwarding behavior.
Next, use sshd -T with a locally supported -C connection description for each case:
/usr/sbin/sshd -T -f /tmp/test-ssh/sshd_config \
-C user=alice,host=client-a.example,addr=198.51.100.10,laddr=192.0.2.10,lport=22The sshd manual defines -T as extended test mode and -C as connection parameters used to apply relevant Match rules. Supported fields include addr, user, host, laddr, lport, and rdomain in current upstream documentation; use only fields confirmed by the installed manual. Extract the values that govern the contract and compare them with the expected matrix. sshd -T -C does not authenticate a user, read a client private key, prove PAM, or establish a forwarding channel.
Store the complete command, output, and exit status for each case. Hold on a nonzero exit, an unsupported parameter, a missing required value, or any mismatch with the contract. Do not reinterpret an unexpected dump as acceptable after the fact; correct the candidate or the written contract and repeat the review.
Build the Per-Connection Policy Matrix
The worksheet below shows expected assertions for an unexecuted example. During a real change, replace the status column with the captured sshd -T output digest, command exit status, and live-test evidence.
Case | Expected effective values | Required fresh assertion | Article status |
alice / source A | password no; pubkey yes; forwarding local | Key login accepted; approved local forward opens | Unexecuted example |
alice / source B | password no; pubkey yes; forwarding no | Key login accepted; forwarding refused by sshd | Unexecuted example |
bob / source A | password no; pubkey yes; forwarding no | Key login accepted; forwarding refused by sshd | Unexecuted example |
bob / source B | account/source admission denied | New login refused by sshd for the intended policy reason | Unexecuted example |
recovery identity | approved key path retained | New recovery login accepted over the approved route | Unexecuted example |
Do not mark a row accepted merely because its expected values look plausible. The article does not claim these rows were executed. A real acceptance record needs the exact command, exit status, captured effective values, candidate digest, and a fresh connection or forwarding attempt that exercises the relevant behavior. A green effective-policy row can still fail live because the account is locked, the shell is unavailable, PAM denies the session, a key file has unsafe ownership, or the network path never reaches the daemon.
If any effective-policy row differs from the contract, hold and repair the candidate before starting a test listener or reloading the service.
Inspect the Client Before Testing the Server
Inspect the client before attributing an outcome to the server. Use the locally installed OpenBSD ssh(1) client manual and ssh -G, when supported, to record the resolved target, port, user, proxy or jump settings, identity files, known-hosts files, and multiplexing options. Confirm that a host alias does not redirect the test to another endpoint. Select one synthetic key explicitly with -i and -o IdentitiesOnly=yes; disable agent selection with a locally supported IdentityAgent setting or a controlled SSH_AUTH_SOCK so an unrelated key cannot succeed. Establish the test server's host key deliberately in a dedicated known_hosts file, then use StrictHostKeyChecking=yes. The ssh_config documentation identifies ask, not no, as the default and explains that yes refuses unknown or changed keys. Never solve a laboratory trust problem with StrictHostKeyChecking=no.
Disable connection reuse with both -o ControlMaster=no and -o ControlPath=none, or the locally documented equivalent. ControlMaster=no alone is insufficient when a configured ControlPath points to an existing master; the client may still attach to that connection. Also neutralize ControlPersist for the dedicated test path and confirm that no wrapper or alias restores a control socket. Record the complete client command and resolved configuration for each case.
A reproducible fresh-test command should pin the destination, port, user, identity, known-hosts database, authentication method, proxy behavior, and multiplexing settings. Keep host-key checking enabled and use a deliberately established host key. The command can be configured to request a new connection, but the acceptance record must still confirm a new TCP connection and handshake in server evidence. If the client exits before key exchange, classify that separately rather than treating it as server-policy enforcement.
Prove the Tests Use Fresh Connections
Confirm freshness on both sides. On the client, verbose output should show a new TCP connection, protocol exchange, and key exchange rather than attachment to a control socket. On the server, correlate the attempt time, source address and port, destination address and port, sshd process or connection identifier, and authentication log. A new child process, connection identifier, or equivalent log sequence is stronger evidence than an old shell that still works. If the normal log level is insufficient on the isolated listener, use a locally supported verbose or debug setting only for the controlled test, avoid logging secret material, and restore the prior level afterward.
The misleading-success control is deliberate. First open a multiplexed connection or forwarding channel under the old policy. Change the candidate so the same operation should be refused, then issue another session or channel request through the existing ControlPath. Because the master connection and its server-side process predate the change, an old tunnel may remain open and a new channel can appear to validate the updated policy even though no new authentication occurred. Capture that behavior as the false-green control, then repeat with ControlPath=none and correlate a new server-side handshake. Only the newly negotiated connection is valid post-change evidence.
Keep the Recovery Session Out of the Test Result
Keep the original administrator session open solely as a rollback route. Label it pre-change, note when it was established, exclude it from the result matrix, and never use a shell, command, or forwarding channel created through it as proof that the new policy permits access. The old session may survive a reload and is valuable for recovery, but each post-change acceptance claim requires a separate newly negotiated connection.
Exercise Allowed and Denied Paths Deliberately
Exercise the allowed and denied paths once per owned test case, observing both client and server evidence. The assertions below are expected outcomes for the proposed lab, not reported observations:
Allowed public-key login: use the designated synthetic key with IdentitiesOnly=yes, PreferredAuthentications=publickey, BatchMode=yes, ControlPath=none, and strict host-key checking. Run a harmless command such as true or id under the synthetic account rather than opening an unnecessary interactive shell. Acceptance requires client exit status 0, the expected command result, and a matching server acceptance record for the new connection and key fingerprint.
Prohibited password method: make one controlled attempt with the synthetic account, explicitly disabling public-key and keyboard-interactive fallbacks as supported by the installed client. Use only the synthetic test password and limit the attempt count; this is not a guessing exercise. Acceptance requires evidence that the server did not offer or accept password authentication under the active policy. A network failure, host-key rejection, or client-side suppression of prompts is not equivalent to a server denial.
Disallowed forwarding: start a new connection with ControlPath=none and request an inert local forward, using ExitOnForwardFailure=yes where supported. Bind a high local port and point it at an owned destination on the test host. Trigger one connection through the local listener so the server receives the channel-open request. Acceptance requires a refusal attributable to sshd, such as an administratively prohibited channel, with a corresponding server record. A remote target connection refusal means the forwarding channel was allowed but the destination did not accept the connection, which is a different result.
Use only owned users, source interfaces, host keys, addresses, and endpoints. Documentation-range addresses in this article are labels, not literal reachable targets. Do not probe public hosts, repeat passwords, or forward traffic to an external service. Stop after the planned attempts, terminate test processes, close inert listeners, and remove synthetic user and host private keys according to the cleanup plan.
Classify the outcome before deciding. Connection refused usually means the network endpoint or listener was unavailable. A timeout may indicate routing, firewall, or address-selection failure. Host-key verification failure means the client rejected server identity before authentication. Permission denied indicates authentication failure but still requires the server reason and the methods actually offered. An administratively prohibited channel indicates forwarding-policy enforcement. A remote target refusal after a channel opened means forwarding was permitted. Do not collapse these distinct mechanisms into a generic failed test.
Record the exact command, client exit status, relevant client diagnostic, server record, connection time, and policy digest for each attempt. Only captured evidence can convert an expected assertion into an accepted result; a successful rerun does not prove that resources or access from the failed attempt were handled correctly.
Stage the Daemon Without Changing the Policy Under Test
Where the owned environment permits it, run an isolated test daemon before touching the primary listener. Prefer a disposable VM with console access and synthetic host keys. An alternate listener on the same host is acceptable only after confirming that the candidate has no conflicting ListenAddress behavior and that host-key paths, privilege requirements, filesystem namespaces, PAM context, security policy, and launch flags match the intended daemon closely enough for the bounded test. It must not replace the production listener or share an ambiguous PID file. One locally reviewed example is:
sudo /usr/sbin/sshd -D -e -f /tmp/test-ssh/sshd_config \
-p 2222 -o PidFile=/tmp/test-sshd.pidThis keeps the process in the foreground, writes diagnostics to standard error, and selects port 2222 through the invocation. Confirm the behavior against the installed manual and candidate directives; command-line port selection can interact with ListenAddress entries. A connection to port 2222 does not satisfy a Match LocalPort 22 condition, and a different local address can change LocalAddress matching. The alternate listener therefore proves only the policy for that alternate endpoint. Record its host keys, PID, log destination, and stop procedure.
Run the same effective-policy and fresh-connection checks against the staged endpoint, including an allowed key login, a prohibited method, and a forwarding assertion. Compare each result with the expected matrix. Then repeat the required cases at the intended local address and port during the controlled maintenance step. A canary on 2222 is useful isolation evidence, but it is not proof of the port-22 policy when LocalPort, LocalAddress, socket activation, firewall rules, or service wrappers differ.
Stop the staged daemon using the recorded process-control method. Confirm that its listener and PID file are gone, archive the candidate manifest and test evidence, and remove synthetic host and user keys according to the lab cleanup plan. Do not record a pass merely because the process started; all required assertions need captures.
Match the Test Environment to the Intended Connection
Align the final acceptance attempt with the intended connection identity: source address, resolved host name if used, destination address, local port, routing domain, account, groups, key, and client method. Repeat any case whose staging endpoint changed a Match input or a surrounding enforcement layer. Avoid a universal systemctl recipe; use the locally verified service, console, and recovery procedure for this host.
Reload the Exact Approved Configuration
Before deployment, compare the candidate manifest and SHA-256 digests with the exact files placed in the active tree, including every Include fragment and service override. Re-run sshd -t and the full sshd -T -C matrix against those bytes using the installed binary and approved invocation flags. Confirm the recovery route and narrow canary scope again. Then use the service environment's documented reload or restart action. The sshd manual states that a SIGHUP causes sshd to reread its configuration by executing itself with the name and options it was started with, but a service manager, socket activation, container supervisor, or wrapper may implement a different operational procedure. Do not substitute a generic service command for local documentation.
Keep the out-of-band route and pre-change session available while the canary set runs. Do not infer reload completion from a command exit alone or from the fact that an old shell remains usable. Confirm the active listener, process invocation, service logs, configuration digests, and at least one required new recovery connection. If the environment requires a restart rather than a reload, follow the locally tested procedure and understand which processes, sockets, and existing sessions it affects. Preserve the recovery boundary until the required new connections have succeeded.
After the service action, repeat the effective-policy matrix and all critical fresh live tests at the actual endpoint. Verify both required access and prohibited behavior. A restored or deployed file is not an accepted service until the daemon has consumed the intended bytes and the required connection behavior has been observed under the approved invocation.
Correlate New Attempts With Server Evidence
Correlate every post-change attempt with a single evidence row. Record the following fields so another reviewer can distinguish policy enforcement from an unrelated client or network failure:
Time, source IP, and destination port of the attempt.
Policy digest: e.g. a hash of the config or a timestamp, so it’s clear which policy version was active.
Client result: exit status and any diagnostic text.
Server log entries: copy the relevant sshd log lines (e.g. from /var/log/auth.log or systemd journal) including acceptance or denial messages.
Use a compact evidence record such as the following. The entries describe required fields and expected categories; they are not observed log output:
Case and time | Connection identity | Policy and client evidence | Server evidence |
alice / source A; recorded UTC time | source, destination address, local port, user, key fingerprint | candidate digest; exact command; exit status; relevant client diagnostic | new connection identifier; authentication result; forwarding result |
bob / source B; recorded UTC time | source, destination address, local port, user | same active digest; exact command; exit status | new connection identifier; explicit policy denial reason |
Use synthetic names in examples and redact real identifiers when operational records require it. The purpose is correlating operational evidence across client command, connection identity, configuration digest, and server decision, not collecting every log line. Preserve enough server wording, process context, and timing to identify the enforcement point and distinguish a policy denial from a missing network path. Never store private keys, passwords, full secret-bearing command lines, or unrelated user data in the article or change ticket.
Separate Enforcement Failure From Missing Evidence
If the expected fresh handshake or server record is missing, hold. The request may have failed before reaching sshd, used the wrong hostname or port, selected the wrong address family, reused a control connection, or been blocked by a firewall, security policy, proxy, or service wrapper. A client-side error without corresponding server evidence does not prove that the new server policy denied the operation. Likewise, a server log with no matching client command may belong to another attempt. Reconcile time, connection identity, and digest before assigning a result.
Mark each row accepted, failed, or inconclusive and record the reason. Inconclusive evidence is not a secure denial and cannot support proceed. Capture the missing evidence, repair the bounded test path or logging path, and rerun only the planned case. Do not turn uncertainty into a pass because the maintenance window is closing.
Restore the Prior State When Acceptance Fails
If a required login fails, a prohibited method succeeds, forwarding exceeds the contract, the active bytes differ from the approved manifest, or evidence remains inconclusive after the maintenance window, use the predefined recovery channel. Stop the staged listener if it contributes ambiguity. Restore the complete prior configuration tree, service overrides, host-key references, and daemon invocation, not a single file. A host-specific example might be:
sudo rsync -a --delete /var/backups/ssh-before/ /etc/ssh/
# Run the host's previously verified reload or restart command.Immediately validate the restored tree with sshd -t, the required sshd -T -C cases, and fresh connections for the recovery identity and affected access paths. Confirm the restored digests, active process arguments, listening address and port, service ownership, and server logs. Recheck both a required allowed path and the important prohibited path so the rollback does not introduce a different exposure. A copied file without a successful service action and fresh access check is not a completed restore. Document the failed acceptance condition, the recovered safe state, and the evidence needed before another change attempt.
Treat restoration as a gated operation. The same recovery dependencies and acceptance gates used in broader recovery planning apply here: console access available, prior tree restored, prior invocation active, syntax valid, effective policy returned, required fresh login re-established, prohibited behavior still denied, and test artifacts cleaned up. Close the incident only after those assertions have evidence and ownership has been handed back to the service operator.
Proceed, Hold or Restore
Use the final matrix to choose one outcome and assign the next action to a named owner:
Condition | Decision and action | Owner |
All required rows match the contract with fresh client and server evidence | Proceed: retain evidence, monitor the canary scope, complete cleanup, and hand off | SSH service owner / change approver |
A required allowed path fails or evidence is incomplete | Hold: preserve the safe state, investigate the exact failed assertion, and rerun the bounded test | SSH service owner |
A prohibited authentication or forwarding path succeeds | Restore: use the recovery route, restore the prior tree and invocation, then revalidate | SSH service owner with security escalation |
Active bytes, invocation, endpoint, or Match inputs differ from the approved test | Hold or restore according to exposure; do not proceed on an untested policy | Change owner and approver |
For an approved change, hand the policy digest, Include manifest, connection matrix, client commands, server evidence, and cleanup status to the service owner and any dependent automation owner. For a hold or restore, update the change or incident record with the exact failed assertion, current safe state, responsible owner, and the minimum evidence required before another attempt.
Reliable remote-access changes depend on Linux administration, networking, security, troubleshooting, backup, and recovery fundamentals. Refonte Learning's System Administration Training & Internship Program presents those foundations through system-configuration topics and practical labs. Review the live program page for current curriculum and admission details, then use an owned practice environment to strengthen the evidence-first habits behind configuration review, controlled testing, and recovery.
The governing invariant is simple: the configuration bytes and daemon invocation that were approved must produce the expected effective policy for each connection identity, and a newly negotiated connection must demonstrate the corresponding allowed or denied behavior. Proceed only when every required row has that evidence and the recovery route remains available. Hold when evidence is missing, the endpoint differs, or the contract is unresolved. Restore when the deployed policy permits a prohibited path, removes required access, or cannot be tied to the approved manifest.
