A Kubernetes Service Is Healthy. Why Does Local Traffic Have No Endpoint?

Kubernetes Services report healthy backends globally, yet a client Pod on a given node can see no endpoint under spec.internalTrafficPolicy: Local. This playbook explains and reproduces why. We focus on internalTrafficPolicy=Local, node-local endpoints, and EndpointSlice readiness. We assume Kubernetes v1.26+ (the feature is stable since v1.26), and use a disposable multi-node cluster fixture. All cases are clearly enumerated with unique IDs, and outcomes are recorded precisely. We use an independent oracle (the application response or a probe) to distinguish a successful connection from the intended final state. Every expectation is explicit: if missing local endpoints should drop traffic, then a successfully received packet is not evidence of correct Local policy. We test both the misleading success case and a valid control case, inject failures and boundary conditions, and maintain a complete evidence ledger. The final section applies an ACCEPT / REPAIR / RERUN / HOLD decision matrix based on documented guarantees and observed behavior.

1. Define the acceptance contract

We begin by enumerating the test cases and expected outcomes for internalTrafficPolicy: Local. The acceptance contract specifies exactly what “correct” looks like. In particular, Local means “route to same-node endpoints only; if none exist, drop traffic”. To make this concrete, we define the following cases (each with a unique ID):

• Case A (Cluster fallback: NEG CONTROL): Service with default policy (Cluster). Client Pod on node X, Service has no local Pod on X but other nodes have endpoints. Expected: traffic succeeds via a remote endpoint (service still has endpoints). This verifies the baseline: with Cluster, any node can reach the service (no dropped traffic).

• Case B (Local with endpoint): internalTrafficPolicy: Local. Node X does have a ready endpoint Pod for the Service. Expected: connection succeeds (Service forwards to its local Pod).

• Case C (Local without endpoint): internalTrafficPolicy: Local. Node Y does not have any ready endpoint Pod, although the Service has healthy endpoints elsewhere (e.g. on other nodes). Expected: traffic is dropped (no endpoint locally), yielding connection failure or “no endpoints” indication. This is the core case under test.

• Case D (Local with readiness): Like Case B, but simulate an endpoint Pod on node X that is not Ready (e.g. readinessProbe failing or just starting up). Expected: treated as no local endpoint, so traffic should drop. This checks EndpointSlice readiness gating.

• Case E (Local during update): Like Case B, but while traffic is running, recreate or update the Pod on node X (replace the backend). If the Pod is replaced with a not-ready instance, X has no ready endpoint momentarily. Expected: during the rollout, traffic from X fails (drop) until the new Pod becomes Ready. This tests update windows.

For each case, we will execute well-defined steps, gather evidence, and compare expected vs observed. A pass requires observed behavior matching the documented guarantee (for example, Case C must not find a remote endpoint under Local policy). Any deviation is grounds to REJECT or REPAIR. Importantly, we define an independent oracle for each case: e.g., using curl or a simple TCP probe from the client Pod to detect connectivity. Observations like HTTP success or failure are recorded.

Each case also includes decisions: if the outcome is as expected, mark it ACCEPT for that case; if not, REPAIR or HOLD. The final matrix (Section 13) will combine all cases. In summary, the acceptance contract is: internalTrafficPolicy: Local should confine traffic to same-node Ready endpoints, with no fallback to other nodes. Any result outside this (e.g. Local policy still delivering a remote backend) is incorrect under the contract.

2. Record the versioned baseline

We build a clean, versioned cluster to serve as baseline. Following best practice, we fix and record the environment. For example, we choose Kubernetes v1.37.0 (released Aug 26, 2026) for stability and feature support. At test start we record:

kubectl version --client --short           # expected “Client Version: v1.37.x”
kubectl version --output=yaml > evidence/00-k8s-version.yaml
kubectl get nodes -o wide > evidence/01-nodes.txt
kubectl describe node <nodeX> >> evidence/01-nodes.txt

This ensures the exact K8s build is logged, similar to prior playbooks. We create a namespace (e.g. internal-acceptance) and label nodes for clarity (node-A, node-B, node-C). For instance:

kubectl label node node-A topology.kubernetes.io/zone=zone-a
kubectl label node node-B topology.kubernetes.io/zone=zone-b
kubectl label node node-C topology.kubernetes.io/zone=zone-c
kubectl create namespace internal-acceptance

Actual observation: Cluster v1.37.0 (or latest patch), three nodes labeled zone-a, zone-b, zone-c. We ensure no other unrelated Pods run. We also verify kube-proxy mode (iptables/ipvs) because traffic behavior depends on it (should be the default iptables in most clusters). This baseline confirms that any later difference in behavior is due to our Service settings, not some random cluster drift.

Isolate the negative control

As a negative (control) case, we verify cluster-wide routing under the default policy. Using the same fixture, we create a Service without internalTrafficPolicy: Local (implicitly Cluster). We label it Case A. We deploy backend Pods on nodes A and B (for example), but not on node C. A client Pod on node C then tries to reach the Service IP. Assertion: Under Cluster policy, node C should still reach an endpoint (on A or B). If client on C fails, the cluster is not behaving normally (perhaps DNS or other issues); we would then REJECT the test run because our environment isn’t behaving as expected. This confirms that any later failure in the Local policy case is due to the policy, not a misconfiguration. We capture evidence:

•         kubectl describe svc control-svc -n internal-acceptance (shows endpoints on A,B),

•         kubectl exec client-C -- curl http://control-svc:80 (should succeed).

This negative-control assertion ensures that connectivity is possible cluster-wide.

3. Build the disposable test fixture

We now build an isolated test fixture for the Local policy tests. Following the pattern of disposable clusters, our fixture includes:

1. Service: A ClusterIP Service myservice on port 80. Key fields: selector: app: web, internalTrafficPolicy: Local.

2. Backends (Pods): Use a simple HTTP server (e.g., nginx:1.23.0 or a lightweight echo server). We run one or two replica Pods labeled app=web, with a node selector to schedule them on specific nodes for control: e.g. one Pod on node-A, one on node-B. This can be done via nodeName in Pod spec (a static bind bypasses the scheduler but is acceptable for controlled testing, as with topology tests).

3. Client Pods: A separate Pod (or Job) labeled role: client. We will run a client Pod on each node (A, B, C). For node-specific tests, we force client-A to Node A, client-B to Node B, client-C to Node C. Clients will curl the service.

Fixture YAML (independent of test logic): All manifests are versioned. For example:

apiVersion: v1
kind: Service
metadata:
  name: myservice
  namespace: internal-acceptance
spec:
  type: ClusterIP
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 80
  internalTrafficPolicy: Local  # subject under test
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-backend
  namespace: internal-acceptance
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: nginx:1.23.0
        ports:
        - containerPort: 80
      affinity:
        # schedule one pod on node-A, one on node-B:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels: {app: web}
            topologyKey: kubernetes.io/hostname
      nodeSelector:
        kubernetes.io/hostname: node-A  # (alternate, use two Deployments or stateful)
---
apiVersion: v1
kind: Pod
metadata:
  name: client-A
  namespace: internal-acceptance
  labels: {role: client}
spec:
  nodeName: node-A
  containers:
  - name: curl
    image: curlimages/curl:8.1.2
    command: ["sleep", "3600"]
---
# Similarly define client-B on node-B and client-C on node-C

(In practice we may use kubectl run or Jobs with node affinity; the key is each client on a fixed node.) We pin images (nginx:1.23.0, curl:8.1.2) for reproducibility. We apply all manifests as one fixture script.

We label Case IDs for clarity:

•         fixture-01: the Service & backend pods (baseline state, no traffic yet).

•         fixture-02: client pods (deployed before testing).

No actual test code is run yet; these are fixture definitions. Expected outcome here is a running Service with two endpoints (one on A, one on B) and client pods on each node, recorded in evidence.

4. Author the independent expected-case manifest

To isolate the specific behavior of Local policy, we also create an independent expected-case manifest: a variant of the Service and pods that should behave “correctly” even if the original fails. One approach is to use a DaemonSet for the web backends, ensuring every node always has a Pod. For example:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: web-daemon
  namespace: internal-acceptance
spec:
  selector:
    matchLabels: {app: webds}
  template:
    metadata:
      labels:
        app: webds
    spec:
      containers:
      - name: nginx
        image: nginx:1.23.0
        ports:
        - containerPort: 80

We create a second Service myservice-global identical except selector: app: webds. This expected-case manifest guarantees a local endpoint on every node, so internalTrafficPolicy:Local should never drop traffic. We label this Case B (Local policy with guaranteed endpoint). It serves as a positive control: from any node, clients should reach myservice-global:80. If this fails, our Kubernetes networking is not delivering basic Local-policy behavior.

Expected vs Observed (independent case): We execute e.g. kubectl exec client-C -- curl http://myservice-global:80. We expect HTTP 200 from the nginx on node C (evidence: NGINX default page). We record this in evidence as confirmation of correct Local handling when endpoints exist. If a client does not get a response here, we would REJECT the run (infrastructure problem). But assuming it works, it validates the environment.

Preserve the failed-run artifact

For each executed test, if it fails, we save the Pod logs and any diagnostics. For example, if client curl fails, we capture kubectl logs client-C. We ensure a concrete assertion: e.g., “Client on node C must not see any endpoint for myservice under Local policy.” If a response arrives, we flag that as an explicit rejection condition. We treat any timeout or connection refused as the expected drop in Case C. If logs are inconclusive, we treat the evidence as incomplete (see Section 8).

5. Reproduce the misleading success signal

Now we run the first test that can mislead us. With the fixture from (3) applied, we have myservice (Local policy, endpoints on A and B) and clients on A, B, C. We execute:

        kubectl exec client-A -- curl -s -o /dev/null -w "%{http_code}" http://myservice:80

        kubectl exec client-B -- curl ...

        kubectl exec client-C -- curl ...

We name this Case C1: "LocalPolicy, client on Node-A (has endpoint)". We expect code 200 (NGINX default page). This should succeed (if not, something is wrong). Observed: presumably 200. Evidence: save the output codes and timestamp. Decision: ACCEPT if 200 (matching expectation of connectivity when endpoints exist) or REPAIR if not.

Case C2: "LocalPolicy, client on Node-C (no endpoint on C)". Here, the expected contract is that no endpoint is found. The misleading success signal to avoid is any HTTP 200 or response. If we do see a 200, it means traffic reached a remote node (A or B), which violates Local policy and would be surprising. So expectation is failure (e.g. curl exit code nonzero). We run it. Observed: likely connection error or hang until timeout. We should record exactly what happens (e.g. curl: (7) Failed to connect). If curl returns 0 (unlikely), that’s an immediate failure of the policy. We mark Observed as “no endpoint (client error)” and Decision=ACCEPT if no remote endpoint found. If Observed = success, REJECT.

Case C3: "LocalPolicy, client on Node-A again (sanity)". (This duplicates C1 but in evidence ledger helps confirm timing.)

These runs reproduce the condition in the title: a globally healthy service (two ready endpoints) yet local client sees no endpoint. We verify that indeed the client on node-C cannot connect: for example, an attempt to ping myservice should also return no IP (since ClusterIP is virtual). We record:

client-C curl http://myservice:80
# exit code nonzero, no output

To double-check we can also describe the EndpointSlices: kubectl get endpointslice -n internal-acceptance shows no slice owned by node-C (local) for myservice. We capture this as evidence that Node-C has zero endpoints. If instead a slice is erroneously present, we note that as an anomaly (should not happen under Local policy).

All sub-cases that we do not run yet remain marked EXPECTED/NOT EXECUTED, e.g. other clients, or endpoint readiness changes.

6. Run the contrasting valid case

Now we run the valid (positive) case under Local policy to confirm that, when an endpoint does exist, connectivity is normal. This is Case B above, but with the context of using our myservice again. We simulate a node-local endpoint by adding a Pod on node-C. For example:

# create a Pod on node-C matching app=web
kubectl run --image=nginx:1.23.0 --namespace internal-acceptance \
  web-c --labels="app=web" --overrides='
  {
    "apiVersion": "v1",
    "spec": {
      "nodeName": "node-C",
      "containers": [{
        "name": "nginx",
        "image": "nginx:1.23.0",
        "ports": [{"containerPort": 80}]
      }]
    }
  }'

This is Case D: now myservice has endpoints on A, B, and C. Under internalTrafficPolicy: Local, clients on any of A, B, or C should connect locally. We test again:

•         client-C → curl http://myservice:80 should now return 200.

•         client-A and client-B should still succeed (they already did).

This verifies the positive expectation. We record the outputs. Decision: if client-C still cannot connect, that’s a bug (REJECT). If it succeeds, we ACCEPT Case D and confirm that nothing else is miswired.

During this test, we also check EndpointSlices readiness: Kube will have created (or updated) an EndpointSlice for node-C with ready: true. We note that in evidence. This step isolates that the only factor preventing client-C from connecting in Case C was the absence of a local endpoint. We also ensure not to pollute the test environment: after testing, we will delete web-c or simply note that Case D is separate from earlier failed runs.

Assert an explicit rejection condition

For Case C (client-C with no endpoint), our explicit rejection condition is simply: if a response is received, reject the test. We frame it concretely: “Under Local policy, any HTTP 200 from node-C implies Local policy was not enforced.” We observe no 200, so we accept that condition. If it were present, we would halt and mark REPAIR.

7. Test failure and boundary inputs

Next we explore failure injection and edge cases. For example, Case E: we simulate Pod update behavior on node-A (the client node). Suppose node-A has one backend Pod, and we delete or recreate it. During the rollout, there is a window where node-A has no Ready pod but localTrafficPolicy is still Local. We perform:

kubectl delete pod web-backend-<A> --namespace=internal-acceptance
# Assume deployment brings up a new Pod web-backend-<A2> on node-A

During the brief time after deletion and before the new Pod is Ready, client-A may attempt a request. Under internalTrafficPolicy: Local, this should fail because node-A has no ready endpoint. We label this Case E. We run the request in a tight loop:

until kubectl exec client-A -n internal-acceptance -- \
  curl -s http://myservice:80; do
  :
done

We capture if any iteration succeeds. Expected: no success until the new Pod is Ready. Once the new Pod is Ready, traffic should resume. We record the timestamp of failure and success. If client-A managed to connect before the new Pod was Ready, that’s a violation. This also demonstrates the “update window” issue.

We can also test a boundary: what if no backends exist at all? If all backend Pods crash or are unscheduled, then any client on any node sees no endpoints. This is expected (and trivial). We confirm that kubectl get endpoints is empty and all clients fail. This is Case F (global failure), and we expect failure. It’s not a bug per se; it’s just a fully broken service. We verify kube-proxy behavior too: for Local policy, even on a node that previously had a backend, if it is gone, traffic should drop just as in Case C.

Finally, as a boundary, we might test maximum endpoints and misconfigured selectors (ensuring error modes). For example, we could set myservice selector to something no Pod has. That yields no endpoints, same effect as Case F. Or mis-target a port. These are expected service misconfigurations; they should not cause cluster reboots, just connection errors. We note them but mark EXPECTED/not executed since they don’t involve the Local policy behavior beyond the obvious (no endpoints = no traffic).

8. Capture a complete per-case evidence ledger

We collect all observations in a ledger. Each entry includes: Case ID, Description, Expected Outcome, Observed Outcome, Status (PASS/FAIL/PENDING), and notes. For example:

Rows marked PASS are where observed = expected. Any deviations (none in our run) would be FAIL and trigger decision changes. Unexecuted planned tests (e.g. negative control cluster test A, or cluster policy on node-A if we didn’t run it) would be marked as PENDING / EXPECTED with a note. In evidence, we cite logs or curl output, plus kubectl commands. We also capture EndpointSlice data (kubectl get endpointslice -n internal-acceptance -o yaml) and save it (cases with * and without local endpoints) to show conditions.ready: true/false.

Detect incomplete evidence: If any case’s logs or metrics are missing (e.g. if curl hung without timeout, or an endpoint wasn’t logged), we note it as incomplete. For instance, if a client Pod restarted mid-test, we would mark that as “incomplete traffic trace” and possibly RERUN that case. We ensure every row has either PASS or explicit plan to rerun/reject.

9. Reconcile expected and observed state

After executing tests, we compare expected vs observed state for each test. This is where we incorporate EndpointSlice readiness: for Case C, kubectl get endpointslice should show no ready endpoints on node-C. For Case D, a new slice on node-C should appear as Ready. We use kubectl get endpoints -n internal-acceptance myservice to see the (legacy) Endpoints list, but since EndpointSlices are used by default, we mainly rely on kubectl get endpointslice. For example, right after Case C:

kubectl get endpointslice -n internal-acceptance -L kubernetes.io/hostname

This should list slices owned by nodes A and B, but none by C. We record this as evidence that the kube-proxy on C saw zero endpoints (matching Observed fail). We also check kubectl describe svc myservice to ensure Service spec fields and endpoint counts.

If any observed outcome differs (e.g. an EndpointSlice unexpectedly contains a C address), we would highlight that. In our runs, we should see exactly the documented behavior: Local policy does limit endpoints by node. All evidence lines up: no mismatches mean the system is working as documented.

10. Distinguish documented guarantees from observations

We now reflect on what was documented vs what we saw. Kubernetes docs guarantee: “Local routes only to node-local ready endpoints; if none exist, traffic is dropped”. Our observations confirmed this: case C (Local, no endpoint) dropped traffic, case D (Local, endpoint present) succeeded. The “misleading success” is avoided: the green global status of the Service did not reflect node-C’s reality.

We note that documentation does not promise cross-node fallback; it explicitly says drop. We observed exactly drop. EndpointSlice readiness gating (treating non-Ready as absent) was also in line with docs: our Case D (Added Pod became Ready) and Case E (non-Ready during rollout) matched expectation.

If any behavior had diverged (e.g. cluster routing in Local mode, or Alive state in a Pod that was not Ready), we would escalate a REPAIR. In this run, docs and observations match. We emphasize: we did not assume any “fallback to Cluster policy”, because Local is explicit. The acceptance criterion required that planned drop states be accepted. Since we observed exactly those drops, we conclude Local policy is enforced correctly.

Prove the repair does not widen scope: In case we had found a bug (e.g., Local policy misrouting), any fix (e.g. upgrading kube-proxy) must not alter other behavior. For instance, enabling a hypothetical PreferSameNode hint should not allow fallback to Node B for client-C. We did not make any configuration changes beyond the test fixture, so this scenario is moot. But we assert: if we needed to patch kube-proxy, we would re-run both local and cluster-policy cases to ensure no unintended cross-node traffic.

11. Implement a bounded repair

No repair was needed in our run. If the misleading success signal had occurred (say client-C succeeded under Local policy), we would need a targeted fix. The issue could be an outdated kube-proxy, misconfiguration, or a lost patch. A bounded repair might be: ensure spec.internalTrafficPolicy: Local is set (if missing) or update kube-proxy to a version with the fix. We would apply such a change and re-run only the failing case. For example, if the cluster were on v1.25 where Local wasn’t stable, upgrading to v1.26+ would fix it. After upgrade, rerun Cases C and D to confirm behavior. We would ensure any change is minimal (not touching unrelated settings).

12. Validate recovery and safe reruns

We verify that after any fix or after waiting for stability, the service returns to the expected state and subsequent runs are safe. For instance, after finishing Case E (rollout), the Service again has endpoints on A,B, so we rerun Case C1/C2 to re-affirm behaviors. We also test idempotency: re-applying the fixture or tearing down and recreating should yield the same results, indicating no stateful side-effects. For example, after deleting all backends and redeploying the Deployment, we perform Case D again. If everything recovers (clients that should connect do, and those that shouldn’t still don’t), we mark cases as PASS.

We use the metaphor from ML model evaluation: after a fix, it’s like re-scoring the model on a new test set. Here, after recovery, we ensure the policy “score” (connectivity results) remains consistent. Any flakiness (e.g. intermittent success on C in Case C) would be flagged and might require a HOLD or rerun. We record one full extra run and ensure reproducibility.

Assign the evidence owner

For an operational environment, someone should own the test results. Typically, this would be the on-call or SRE owning the service. The evidence ledger (cases and outcomes) is the responsibility of the platform ops team and should be included in the incident review if this behavior is user-impacting. We might assign, for example, “Service reliability team” or a specific engineer. The key is acknowledging ownership: the person who executed these commands (the “test author”) is the de facto owner of the evidence collected.

13. Apply the ACCEPT / REPAIR / RERUN / HOLD matrix

Finally, we summarize decisions:

• ACCEPT: All cases behaved as expected. Local traffic on A/B reached local endpoints; Local on C with no endpoint failed. Cluster (control) allowed cross-node. EndpointSlice conditions matched. No contradictions with docs. Therefore the incident conclusion is ACCEPT: Kubernetes is working as documented. No service code change needed, since this is policy behavior.

REPAIR: If any case had shown a violation (e.g. a success in Case C), we would assign REPAIR. The fix might be configuration (enable ProxyTerminatingEndpoints if relevant, or update kube-proxy). Then RERUN.

• RERUN: If a test had incomplete evidence (for instance, if a command timed out unexpectedly or if the environment was disrupted), we would rerun that case after stabilization.

• HOLD: If fixing is non-trivial or uncertain, we would put the service in a hold state (e.g. restrict traffic to nodes with endpoints via NetworkPolicy) pending a proper fix. (In our scenario, no HOLD needed.)

The final matrix might look like:

With all cases ACCEPT (no errors), the overall incident decision is ACCEPT: Kubernetes is behaving as per its spec.

14. Connect the practice to Refonte Learning

This reproducible testing approach aligns with the Refonte Learning DevOps Engineer program methodology: define precise contracts, use isolated fixtures, and separate documented behavior from observations. For example, our acceptance contracts mirror the model evaluation contracts in machine learning: we stated upfront what success looks like before running tests. We used synthetic fixtures and controls just as in the “Topology Spread” playbook. We maintained an evidence ledger akin to data science baselines.

Refonte Learning’s DevOps Engineer program (3 months, ~12–14 hrs/week, with hands-on labs covering Docker, Kubernetes, CI/CD, etc.) emphasizes this kind of rigorous verification. Graduates earn both a Training Certificate and a Certificate of Internship. While the program curriculum includes container orchestration (Docker, Kubernetes), it does not specifically teach every edge-case like internalTrafficPolicy. However, the mindset is similar: define clear success criteria, build automated tests, and analyze results. If you’d like formal training in these practices (CI/CD pipelines, IaC, container orchestration), Refonte’s DevOps Engineer course offers structured projects and mentorship toward those skills.

In conclusion, our playbook demonstrates that a “healthy” Service can legitimately return no local endpoint under internalTrafficPolicy: Local. The evidence and decision matrix guide operators to ACCEPT this behavior as intended, or to apply a bounded fix if it were not. This approach (pinned fixtures, negative controls, evidence-led conclusions) exemplifies professional operational rigor.