Kubernetes Health Checks: Liveness, Readiness, and Startup Probes Explained

Kubernetes Health Checks: Liveness, Readiness, and Startup Probes Explained - Build Archive

Understanding how the control plane actually schedules and tracks pods explains what a probe result feeds into. The three Kubernetes probe types get explained by their names as if the names were self-explanatory — liveness, readiness, startup — and the actual behavior each one triggers on failure is different enough that mixing them up doesn’t just misconfigure a health check, it produces specific, recognizable incidents. Worth walking through what each one actually does, not just what it’s called.

Readiness: “Don’t Send Me Traffic Right Now”

A failing readiness probe removes the pod from a Service’s endpoints — it keeps running exactly as it is, just stops receiving new traffic until the probe passes again:

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  periodSeconds: 5
  failureThreshold: 3

This is the correct tool for temporary, self-recovering unavailability: a service still warming up its connection pool at startup, or briefly overloaded and deliberately shedding new traffic while it works through a backlog. The pod isn’t broken — it’s saying “not right now” — and Kubernetes respects that without taking any destructive action. A failing readiness check with no accompanying liveness failure should never restart anything; if you see a pod cycling on a readiness-only failure, something is misconfigured.

Liveness: “Kill Me and Start Over”

A failing liveness probe does something categorically more drastic — Kubernetes kills the container and restarts it, on the theory that whatever’s wrong won’t fix itself and a clean restart is the safest recovery:

livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  periodSeconds: 10
  failureThreshold: 3
  timeoutSeconds: 2

This is the right tool for exactly one class of problem: a deadlock, a wedged event loop, a leaked-memory state the process genuinely can’t recover from on its own without a restart. It’s the wrong tool for a slow downstream dependency — and this is the most common real-world liveness misconfiguration, worth naming specifically: a liveness endpoint that checks the database connection will restart the application the moment the database is slow, even though the application itself is completely healthy and a restart does nothing to fix a slow database. Restarting a healthy process because something it depends on is having a bad moment doesn’t just fail to help — the fresh instance’s own startup (re-establishing connections, warming caches) adds new load onto the exact downstream system that was already struggling, at the worst possible time.

The rule that resolves this cleanly: liveness should check “is this process itself broken” — an internal deadlock, an unrecoverable internal state — and nothing external. Readiness should check “can this process currently serve a request well,” including the health of things it depends on. Conflating the two into one endpoint both probes hit is the single most common Kubernetes health-check mistake, and it’s the direct cause of the specific, recognizable incident pattern where a slow database turns into a full application restart storm across every pod simultaneously.

Startup: The Probe That Exists to Stop the Other Two From Firing Too Early

Before startup probes existed as a separate mechanism, an application with a genuinely slow boot — loading a large ML model, warming an in-memory cache from a cold store — had an ugly choice: set liveness’s initialDelaySeconds generously long to survive the slow boot, which also means a genuine post-startup deadlock goes undetected for that entire delay window every time. The startup probe separates these concerns:

startupProbe:
  httpGet:
    path: /health/started
    port: 8080
  periodSeconds: 5
  failureThreshold: 60   # up to 5 minutes to start, checked every 5s

livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  periodSeconds: 10
  failureThreshold: 3    # can afford to be tight — startup is already handled

While the startup probe is still failing, liveness and readiness are held off entirely — they don’t even begin evaluating. The moment startup succeeds once, it stops running for the rest of the pod’s life, and liveness takes over with whatever tight, responsive threshold you’d actually want for catching a real post-startup deadlock quickly. This is the fix for “slow to start” and “quick to detect once running” being in genuine tension when they’re forced to share one probe’s configuration — they no longer have to.

A Concrete Failure Case Worth Walking Through

A deployment updates, the new pods start, and the rollout stalls with pods stuck in a restart loop that never stabilizes. The diagnostic path that actually finds the cause fast:

$ kubectl describe pod api-7d9f8b6c-x2n4p
  Liveness probe failed: HTTP probe failed with statuscode: 503
  Back-off restarting failed container

$ kubectl logs api-7d9f8b6c-x2n4p --previous
  connecting to database... (attempt 4, retrying)

--previous is the detail that matters here — it fetches logs from the container instance that just got killed, not the fresh one that replaced it and has barely started logging anything yet. In this shape of failure, the previous instance’s logs almost always show it was in the middle of a slow-but-legitimate operation (reconnecting to a database, waiting on a slow dependency) when liveness killed it — which is the fingerprint of a liveness check that’s testing an external dependency instead of the process’s own internal health, restarting a pod that wasn’t actually broken and would have recovered on its own if left alone.

One More Interaction Worth Knowing: Probes Don’t Pause During Shutdown

A detail that catches people during a rollout specifically: when a pod receives SIGTERM as part of a normal, voluntary termination (a rolling update, a scale-down), readiness and liveness probes keep running right up until the container actually exits — Kubernetes doesn’t automatically treat “shutting down on purpose” as a reason to stop checking. A container that stops responding to HTTP the moment it receives SIGTERM, while still finishing in-flight work for a few more seconds, will fail its next readiness check during that window and get correctly pulled from load balancing — which is desired — but can also fail liveness if the shutdown takes longer than the liveness timeout, triggering a forceful kill (SIGKILL) that cuts off graceful in-flight request handling it was already in the middle of doing correctly. The fix is making sure terminationGracePeriodSeconds and the liveness probe’s own timing are set with this overlap in mind, not just tuned independently for the steady-state running case.

The Checklist

  • Liveness checks internal process health only — no database, no downstream API calls in that endpoint’s logic.
  • Readiness checks whether this instance can serve a request well right now, dependencies included — and failing it never restarts anything, just pulls the pod from traffic temporarily.
  • Startup probes exist specifically so a slow boot and a tight liveness threshold don’t have to compromise against each other.
  • kubectl logs --previous is the fastest way to confirm whether a restart loop’s root cause was actually inside the process or in something external it was waiting on.

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *